[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:46:26 dist-test-slave-blv7 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:46:26 dist-test-slave-blv7 krb5kdc[395](info): set up 2 sockets
Sep 29 20:46:26 dist-test-slave-blv7 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 20:46:27.667279 388 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:46:27 dist-test-slave-blv7 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:46:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:46:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:46:27.823454 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40283
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:28.129155 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:28.129418 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:28.129540 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:28.137024 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:46:28.137117 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:28.137185 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:28.137261 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:46:28.137321 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:46:28.146804 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:28.148087 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:28.149335 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:28.161006 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:28.161062 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:28.161062 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:28.161300 408 server_base.cc:1034] running on GCE node
I20260929 20:46:28.162339 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:28.163735 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:28.164924 408 hybrid_clock.cc:648] HybridClock initialized: now 1790714788164852 us; error 58 us; skew 500 ppm
I20260929 20:46:28.165228 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:28 dist-test-slave-blv7 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:28.174796 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:46:28.176339 408 webserver.cc:460] Webserver started at http://127.0.97.62:38665/ using document root <none> and password file <none>
I20260929 20:46:28.176996 408 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:28.177103 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:28.177372 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:28.180225 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "1252a2a712db41788c39fdbdf6d9c27a"
format_stamp: "Formatted at 2026-09-29 20:46:28 on dist-test-slave-blv7"
server_key: "49790d0a28918c331005a07d5d7ff051"
server_key_iv: "bb4a926fe2423fef908895062b4ef4d6"
server_key_version: "encryptionkey@0"
I20260929 20:46:28.180968 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "1252a2a712db41788c39fdbdf6d9c27a"
format_stamp: "Formatted at 2026-09-29 20:46:28 on dist-test-slave-blv7"
server_key: "49790d0a28918c331005a07d5d7ff051"
server_key_iv: "bb4a926fe2423fef908895062b4ef4d6"
server_key_version: "encryptionkey@0"
I20260929 20:46:28.185721 408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 20:46:28.188953 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:28.190109 408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 20:46:28.190259 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "1252a2a712db41788c39fdbdf6d9c27a"
format_stamp: "Formatted at 2026-09-29 20:46:28 on dist-test-slave-blv7"
server_key: "49790d0a28918c331005a07d5d7ff051"
server_key_iv: "bb4a926fe2423fef908895062b4ef4d6"
server_key_version: "encryptionkey@0"
I20260929 20:46:28.190420 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:28.204267 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:28.207845 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:28.208086 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:28.225798 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40283
I20260929 20:46:28.225817 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40283 every 8 connection(s)
I20260929 20:46:28.227150 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:46:28.230592 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:28.237610 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 408
I20260929 20:46:28.237864 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:46:28.239674 388 external_mini_cluster.cc:1428] Setting key 6353272002bba6193a2f8a577755da7b
I20260929 20:46:28.240226 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a: Bootstrap starting.
I20260929 20:46:28.243775 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:28.244951 475 log.cc:824] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:46:28.247699 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a: No bootstrap required, opened a new log
I20260929 20:46:28.256120 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1252a2a712db41788c39fdbdf6d9c27a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40283 } }
I20260929 20:46:28.256562 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:28.256671 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1252a2a712db41788c39fdbdf6d9c27a, State: Initialized, Role: FOLLOWER
I20260929 20:46:28.257212 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1252a2a712db41788c39fdbdf6d9c27a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40283 } }
I20260929 20:46:28.257417 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:46:28.257539 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:46:28.257701 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:28.259982 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1252a2a712db41788c39fdbdf6d9c27a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40283 } }
I20260929 20:46:28.260542 475 leader_election.cc:302] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1252a2a712db41788c39fdbdf6d9c27a; no voters:
I20260929 20:46:28.261574 475 leader_election.cc:288] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:46:28.261688 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:46:28.262651 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a [term 1 LEADER]: Becoming Leader. State: Replica: 1252a2a712db41788c39fdbdf6d9c27a, State: Running, Role: LEADER
I20260929 20:46:28.263247 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1252a2a712db41788c39fdbdf6d9c27a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40283 } }
I20260929 20:46:28.263538 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:46:28.265695 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1252a2a712db41788c39fdbdf6d9c27a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1252a2a712db41788c39fdbdf6d9c27a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40283 } } }
I20260929 20:46:28.266012 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a [sys.catalog]: This master's current role is: LEADER
I20260929 20:46:28.267046 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1252a2a712db41788c39fdbdf6d9c27a. Latest consensus state: current_term: 1 leader_uuid: "1252a2a712db41788c39fdbdf6d9c27a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1252a2a712db41788c39fdbdf6d9c27a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40283 } } }
I20260929 20:46:28.267403 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a [sys.catalog]: This master's current role is: LEADER
I20260929 20:46:28.268893 485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 29 20:46:28 dist-test-slave-blv7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:28.275889 485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:46:28.284329 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:28.242891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36352 (local address 127.0.97.62:40283)
0929 20:46:28.243625 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:28.243636 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:28.246144 (+ 2508us) server_negotiation.cc:407] Connection header received
0929 20:46:28.255533 (+ 9389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:28.255558 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:28.256023 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:28.256434 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:28.259386 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:28.261416 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:28.263770 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:28.264783 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:28.264984 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:28.272578 (+ 7594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:28.272590 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:28.272612 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:28.272665 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:28.277256 (+ 4591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:28.278232 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:28.278240 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:28.278248 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:28.278441 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:28.278996 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:28.279001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:28.279005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:28.279631 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:28.279958 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:28.282605 (+ 2647us) server_negotiation.cc:299] Negotiation successful
0929 20:46:28.283002 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":356,"threads_started":1}
I20260929 20:46:28.295559 485 catalog_manager.cc:1347] Generated new cluster ID: 3e36607b5dac4b3990752e6ad4c3ac16
I20260929 20:46:28.295665 485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:46:28.312211 485 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:46:28.313961 485 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:46:28.322387 485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1252a2a712db41788c39fdbdf6d9c27a: Generated new TSK 0
I20260929 20:46:28.323354 485 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:46:28.362146 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40283
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:28.679423 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:28.679695 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:28.679840 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:28.687335 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:28.687443 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:28.687608 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:46:28.697124 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:28.698305 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:28.699630 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:28.711786 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:28.711813 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:28.712002 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:28.712531 501 server_base.cc:1034] running on GCE node
I20260929 20:46:28.713039 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:28.713821 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:28.715001 501 hybrid_clock.cc:648] HybridClock initialized: now 1790714788714952 us; error 43 us; skew 500 ppm
I20260929 20:46:28.715343 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:28 dist-test-slave-blv7 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:28.718186 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:46:28.719470 501 webserver.cc:460] Webserver started at http://127.0.97.1:46137/ using document root <none> and password file <none>
I20260929 20:46:28.720064 501 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:28.720152 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:28.720387 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:28.723053 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "bbc0668050fb4969b34b3b204c311212"
format_stamp: "Formatted at 2026-09-29 20:46:28 on dist-test-slave-blv7"
server_key: "0428aa6b2abdb99e3631959cd27010e5"
server_key_iv: "9991c01699848c62fb7b8a329e287f55"
server_key_version: "encryptionkey@0"
I20260929 20:46:28.724191 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bbc0668050fb4969b34b3b204c311212"
format_stamp: "Formatted at 2026-09-29 20:46:28 on dist-test-slave-blv7"
server_key: "0428aa6b2abdb99e3631959cd27010e5"
server_key_iv: "9991c01699848c62fb7b8a329e287f55"
server_key_version: "encryptionkey@0"
I20260929 20:46:28.728720 501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 20:46:28.731729 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:28.732887 501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 20:46:28.733000 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "bbc0668050fb4969b34b3b204c311212"
format_stamp: "Formatted at 2026-09-29 20:46:28 on dist-test-slave-blv7"
server_key: "0428aa6b2abdb99e3631959cd27010e5"
server_key_iv: "9991c01699848c62fb7b8a329e287f55"
server_key_version: "encryptionkey@0"
I20260929 20:46:28.733140 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:28.744587 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:28.748054 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:28.748265 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:28.749460 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:46:28.750604 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:46:28.750670 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:28.750751 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:46:28.750789 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:28.780944 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44425
I20260929 20:46:28.780985 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44425 every 8 connection(s)
I20260929 20:46:28.782094 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:46:28.786355 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 501
I20260929 20:46:28.786530 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:46:28.787087 388 external_mini_cluster.cc:1428] Setting key 2e028041009793b41c1bbfb6f85a3acf
Sep 29 20:46:28 dist-test-slave-blv7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:46:28.801460 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:28.784454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38805 (local address 127.0.97.62:40283)
0929 20:46:28.784652 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:28.784658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:28.785707 (+ 1049us) server_negotiation.cc:407] Connection header received
0929 20:46:28.787602 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:28.787608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:28.787680 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:28.787762 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:28.789884 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:28.790957 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:28.792436 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:28.793027 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:28.793138 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:28.796310 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:28.796315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:28.796319 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:28.796348 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:28.798659 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:28.799371 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:28.799377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:28.799380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:28.799513 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:28.799889 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:28.799894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:28.799898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:28.800154 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:28.800277 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:28.800948 (+ 671us) server_negotiation.cc:299] Negotiation successful
0929 20:46:28.801156 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:46:28.802410 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:28.784919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40283 (local address 127.0.97.1:38805)
0929 20:46:28.785568 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:28.785603 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:28.787307 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:28.787992 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:28.788002 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:28.789173 (+ 1171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:28.789714 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:28.789731 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:28.791121 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:28.791126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:28.792299 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:28.792310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:28.793246 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:28.793254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:28.793470 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:28.794084 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:28.794105 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:28.796104 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:28.798778 (+ 2674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:28.798787 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:28.798802 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:28.799269 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:28.799607 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:28.799612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:28.799616 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:28.799781 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:28.800270 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:28.800277 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:28.800609 (+ 332us) client_negotiation.cc:769] Sending connection context
0929 20:46:28.800849 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 20:46:28.801328 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":245,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 20:46:28.803874 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40283
I20260929 20:46:28.804195 629 heartbeater.cc:464] Registering TS with master...
I20260929 20:46:28.805028 629 heartbeater.cc:511] Master 127.0.97.62:40283 requested a full tablet report, sending...
I20260929 20:46:28.807229 440 ts_manager.cc:194] Registered new tserver with Master: bbc0668050fb4969b34b3b204c311212 (127.0.97.1:44425)
I20260929 20:46:28.808583 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:46:28.822443 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40283
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:29.127848 634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:29.128111 634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:29.128209 634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:29.135392 634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:29.135484 634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:29.135598 634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:46:29.144811 634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:29.145987 634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:29.147215 634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:29.159307 639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:29.159226 642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:29.159197 640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:29.159796 634 server_base.cc:1034] running on GCE node
I20260929 20:46:29.160357 634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:29.161249 634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:29.162453 634 hybrid_clock.cc:648] HybridClock initialized: now 1790714789162410 us; error 46 us; skew 500 ppm
I20260929 20:46:29.162737 634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:29 dist-test-slave-blv7 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:29.165660 634 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:46:29.166935 634 webserver.cc:460] Webserver started at http://127.0.97.2:43455/ using document root <none> and password file <none>
I20260929 20:46:29.167563 634 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:29.167655 634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:29.167888 634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:29.171068 634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "c1dcaa9d7a9947d9b872ff55230693b2"
format_stamp: "Formatted at 2026-09-29 20:46:29 on dist-test-slave-blv7"
server_key: "13cfbf5f1a9e651d9fbf701bedda2408"
server_key_iv: "5f056c00c7c29a10e4da474b20925047"
server_key_version: "encryptionkey@0"
I20260929 20:46:29.171742 634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c1dcaa9d7a9947d9b872ff55230693b2"
format_stamp: "Formatted at 2026-09-29 20:46:29 on dist-test-slave-blv7"
server_key: "13cfbf5f1a9e651d9fbf701bedda2408"
server_key_iv: "5f056c00c7c29a10e4da474b20925047"
server_key_version: "encryptionkey@0"
I20260929 20:46:29.176283 634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 20:46:29.179322 649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:29.180507 634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 20:46:29.180625 634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "c1dcaa9d7a9947d9b872ff55230693b2"
format_stamp: "Formatted at 2026-09-29 20:46:29 on dist-test-slave-blv7"
server_key: "13cfbf5f1a9e651d9fbf701bedda2408"
server_key_iv: "5f056c00c7c29a10e4da474b20925047"
server_key_version: "encryptionkey@0"
I20260929 20:46:29.180768 634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:29.201460 634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:29.205997 634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:29.206212 634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:29.207067 634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:46:29.208233 634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:46:29.208302 634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:29.208392 634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:46:29.208431 634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:29.241331 634 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33371
I20260929 20:46:29.241380 761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33371 every 8 connection(s)
I20260929 20:46:29.242743 634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:46:29.246310 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 634
I20260929 20:46:29.246479 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:46:29.246968 388 external_mini_cluster.cc:1428] Setting key 39e5957530b44f37b5955a31c7f00e22
Sep 29 20:46:29 dist-test-slave-blv7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:29.262497 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:29.245806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54345 (local address 127.0.97.62:40283)
0929 20:46:29.246015 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:29.246021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:29.247318 (+ 1297us) server_negotiation.cc:407] Connection header received
0929 20:46:29.249436 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:29.249441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:29.249514 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:29.249600 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:29.250999 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:29.251730 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:29.253170 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:29.253584 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:29.253696 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:29.257011 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:29.257016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:29.257020 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:29.257056 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:29.259532 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:29.260339 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:29.260343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:29.260345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:29.260488 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:29.260998 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:29.261001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:29.261003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:29.261252 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:29.261353 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:29.262120 (+ 767us) server_negotiation.cc:299] Negotiation successful
0929 20:46:29.262271 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:46:29.263722 764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:29.246351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40283 (local address 127.0.97.2:54345)
0929 20:46:29.247156 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:29.247198 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:29.249196 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:29.249829 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:29.249838 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:29.250324 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:29.250843 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:29.250859 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:29.251876 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:29.251880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:29.253044 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:29.253055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:29.253787 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:29.253792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:29.253989 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:29.254585 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:29.254606 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:29.256822 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:29.259655 (+ 2833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:29.259664 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:29.259681 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:29.260221 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:29.260588 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:29.260593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:29.260596 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:29.260876 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:29.261364 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:29.261371 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:29.261770 (+ 399us) client_negotiation.cc:769] Sending connection context
0929 20:46:29.262081 (+ 311us) client_negotiation.cc:240] Negotiation successful
0929 20:46:29.262596 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":310,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 20:46:29.265180 762 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40283
I20260929 20:46:29.265513 762 heartbeater.cc:464] Registering TS with master...
I20260929 20:46:29.266302 762 heartbeater.cc:511] Master 127.0.97.62:40283 requested a full tablet report, sending...
I20260929 20:46:29.267766 440 ts_manager.cc:194] Registered new tserver with Master: c1dcaa9d7a9947d9b872ff55230693b2 (127.0.97.2:33371)
I20260929 20:46:29.268587 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:46:29.282078 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40283
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:29.586445 767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:29.586694 767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:29.586792 767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:29.593741 767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:29.593837 767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:29.593958 767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:46:29.604029 767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:29.605159 767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:29.606367 767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:29.618305 773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:29.618480 775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:29.618971 772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:29.620036 767 server_base.cc:1034] running on GCE node
I20260929 20:46:29.620548 767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:29.621316 767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:29.622520 767 hybrid_clock.cc:648] HybridClock initialized: now 1790714789622483 us; error 32 us; skew 500 ppm
I20260929 20:46:29.622769 767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:29 dist-test-slave-blv7 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:29.625603 767 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:46:29.626889 767 webserver.cc:460] Webserver started at http://127.0.97.3:34721/ using document root <none> and password file <none>
I20260929 20:46:29.627552 767 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:29.627642 767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:29.627875 767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:29.630466 767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "9761164025f64b1fb71de7ecc7804060"
format_stamp: "Formatted at 2026-09-29 20:46:29 on dist-test-slave-blv7"
server_key: "b1996cf1720bb8029603f7e8fc2c9359"
server_key_iv: "01c81fb006c7f4b92dcee02ba89ab19b"
server_key_version: "encryptionkey@0"
I20260929 20:46:29.631187 767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9761164025f64b1fb71de7ecc7804060"
format_stamp: "Formatted at 2026-09-29 20:46:29 on dist-test-slave-blv7"
server_key: "b1996cf1720bb8029603f7e8fc2c9359"
server_key_iv: "01c81fb006c7f4b92dcee02ba89ab19b"
server_key_version: "encryptionkey@0"
I20260929 20:46:29.636037 767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:46:29.639351 782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:29.640614 767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:46:29.640746 767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "9761164025f64b1fb71de7ecc7804060"
format_stamp: "Formatted at 2026-09-29 20:46:29 on dist-test-slave-blv7"
server_key: "b1996cf1720bb8029603f7e8fc2c9359"
server_key_iv: "01c81fb006c7f4b92dcee02ba89ab19b"
server_key_version: "encryptionkey@0"
I20260929 20:46:29.640908 767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:29.660563 767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:29.664489 767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:29.664716 767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:29.665748 767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:46:29.667062 767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:46:29.667145 767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:29.667234 767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:46:29.667274 767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:29.697438 767 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36241
I20260929 20:46:29.697474 894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36241 every 8 connection(s)
I20260929 20:46:29.698594 767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:46:29.703676 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 767
I20260929 20:46:29.703859 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:46:29.704332 388 external_mini_cluster.cc:1428] Setting key 9bb346db58219228bc29ddc2d606b973
Sep 29 20:46:29 dist-test-slave-blv7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:29.717285 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:29.701300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34541 (local address 127.0.97.62:40283)
0929 20:46:29.701464 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:29.701469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:29.702556 (+ 1087us) server_negotiation.cc:407] Connection header received
0929 20:46:29.704660 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:29.704664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:29.704727 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:29.704818 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:29.706449 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:29.707461 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:29.709137 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:29.709532 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:29.709625 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:29.712733 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:29.712738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:29.712741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:29.712770 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:29.714897 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:29.715549 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:29.715553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:29.715555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:29.715656 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:29.716034 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:29.716037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:29.716039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:29.716272 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:29.716370 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:29.716911 (+ 541us) server_negotiation.cc:299] Negotiation successful
0929 20:46:29.717066 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:46:29.718165 897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:29.701761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40283 (local address 127.0.97.3:34541)
0929 20:46:29.702412 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:29.702452 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:29.704378 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:29.705076 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:29.705089 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:29.705603 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:29.706281 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:29.706307 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:29.707600 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:29.707605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:29.708989 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:29.709000 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:29.709704 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:29.709709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:29.709899 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:29.710480 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:29.710505 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:29.712557 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:29.715009 (+ 2452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:29.715048 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:29.715065 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:29.715467 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:29.715773 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:29.715777 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:29.715779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:29.715949 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:29.716357 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:29.716362 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:29.716633 (+ 271us) client_negotiation.cc:769] Sending connection context
0929 20:46:29.716843 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 20:46:29.717176 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":230,"threads_started":1}
I20260929 20:46:29.719539 895 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40283
I20260929 20:46:29.719790 895 heartbeater.cc:464] Registering TS with master...
I20260929 20:46:29.720911 895 heartbeater.cc:511] Master 127.0.97.62:40283 requested a full tablet report, sending...
I20260929 20:46:29.722155 439 ts_manager.cc:194] Registered new tserver with Master: 9761164025f64b1fb71de7ecc7804060 (127.0.97.3:36241)
I20260929 20:46:29.722929 439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34541
I20260929 20:46:29.732421 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:46:29 dist-test-slave-blv7 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714789, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:46:29 dist-test-slave-blv7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714789, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:46:29.772816 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:29.757958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34878 (local address 127.0.97.1:44425)
0929 20:46:29.758492 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:29.758502 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:29.758618 (+ 116us) server_negotiation.cc:407] Connection header received
0929 20:46:29.758790 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:29.758797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:29.759040 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:29.759187 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:29.760522 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:29.761521 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:29.762744 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:29.763303 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:29.763426 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:29.767519 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:29.767529 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:29.767544 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:29.767584 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:29.770209 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:29.770839 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:29.770846 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:29.770848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:29.770964 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:29.771367 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:29.771372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:29.771375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:29.771696 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:29.771991 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:29.772256 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 20:46:29.772496 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":252,"threads_started":1}
W20260929 20:46:29.773972 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34878
Sep 29 20:46:29 dist-test-slave-blv7 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:46:29 dist-test-slave-blv7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:46:29.811224 629 heartbeater.cc:503] Master 127.0.97.62:40283 was elected leader, sending a full tablet report...
I20260929 20:46:29.815703 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:29.802609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34890 (local address 127.0.97.1:44425)
0929 20:46:29.802799 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:29.802806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:29.803082 (+ 276us) server_negotiation.cc:407] Connection header received
0929 20:46:29.803263 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:29.803269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:29.803338 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:29.803421 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:29.804647 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:29.805328 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:29.806492 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:29.806897 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:29.807001 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:29.810514 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:29.810519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:29.810522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:29.810552 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:29.813436 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:29.813994 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:29.813999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:29.814002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:29.814112 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:29.814446 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:29.814451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:29.814455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:29.814694 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:29.814906 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:29.815203 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 20:46:29.815404 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:46:29.816969 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 501
I20260929 20:46:29.828784 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:29.817841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34900 (local address 127.0.97.1:44425)
0929 20:46:29.818026 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:29.818031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:29.818230 (+ 199us) server_negotiation.cc:407] Connection header received
0929 20:46:29.818351 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:29.818354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:29.818406 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:29.818473 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:29.819591 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:29.820241 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:29.821360 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:29.821732 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:29.821817 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:29.824161 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:29.824165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:29.824168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:29.824196 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:29.826619 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:29.827223 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:29.827227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:29.827230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:29.827370 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:29.827674 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:29.827678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:29.827680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:29.827930 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:29.828098 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:29.828305 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 20:46:29.828457 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:46:29.829452 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:46:29.909292 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 501
I20260929 20:46:29.922631 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 634
Sep 29 20:46:29 dist-test-slave-blv7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:46:29.936793 909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:29.923417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36452 (local address 127.0.97.2:33371)
0929 20:46:29.923936 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:29.923943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:29.923970 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:46:29.924030 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:29.924036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:29.924238 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:29.924369 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:29.925434 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:29.926452 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:29.927709 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:29.928240 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:29.928352 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:29.932026 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:29.932037 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:29.932052 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:29.932089 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:29.934659 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:29.935208 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:29.935215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:29.935217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:29.935345 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:29.935648 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:29.935651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:29.935653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:29.935946 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:29.936153 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:29.936344 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 20:46:29.936525 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":251,"threads_started":1}
I20260929 20:46:29.937925 757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:46:30.017213 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 634
I20260929 20:46:30.034485 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 767
Sep 29 20:46:30 dist-test-slave-blv7 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:46:30.049032 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:30.035379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33800 (local address 127.0.97.3:36241)
0929 20:46:30.035941 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:30.035948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:30.035977 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:46:30.036034 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:30.036041 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:30.036248 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:30.036374 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:30.037466 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:30.038475 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:30.039739 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:30.040369 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:30.040478 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:30.044197 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:30.044208 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:30.044222 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:30.044253 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:30.046777 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:30.047454 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:30.047460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:30.047463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:30.047573 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:30.047838 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:30.047841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:30.047843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:30.048157 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:30.048375 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:30.048580 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 20:46:30.048757 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":263,"threads_started":1}
I20260929 20:46:30.050173 890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:46:30.129141 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 767
I20260929 20:46:30.142105 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 408
I20260929 20:46:30.143193 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:46:30.247859 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 408
2026-09-29T20:46:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3573 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:46:32 dist-test-slave-blv7 krb5kdc[916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:46:32 dist-test-slave-blv7 krb5kdc[916](info): set up 2 sockets
Sep 29 20:46:32 dist-test-slave-blv7 krb5kdc[916](info): commencing operation
krb5kdc: starting...
W20260929 20:46:33.546636 388 mini_kdc.cc:121] Time spent starting KDC: real 3.262s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260929 20:46:33.699970 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.153s user 0.001s sys 0.003s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260929 20:46:33.959712 388 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.163s user 0.000s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/test-user.keytab.
W20260929 20:46:34.487071 388 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.527s user 0.001s sys 0.002s
Sep 29 20:46:34 dist-test-slave-blv7 krb5kdc[916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:46:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:46:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
W20260929 20:46:35.425091 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.62: real 0.898s user 0.003s sys 0.003s
I20260929 20:46:35.426095 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46257
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:35.738512 929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:35.738790 929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:35.738909 929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:35.746024 929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:46:35.746120 929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:35.746189 929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:35.746268 929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:46:35.746325 929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:46:35.755956 929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:35.757251 929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:35.758541 929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:35.769564 937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:35.769662 934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:35.769886 929 server_base.cc:1034] running on GCE node
W20260929 20:46:35.769642 935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:35.770925 929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:35.772261 929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:35.773474 929 hybrid_clock.cc:648] HybridClock initialized: now 1790714795773430 us; error 42 us; skew 500 ppm
I20260929 20:46:35.773789 929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:35 dist-test-slave-blv7 krb5kdc[916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:35.776662 929 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:46:35.778110 929 webserver.cc:460] Webserver started at http://127.0.97.62:45003/ using document root <none> and password file <none>
I20260929 20:46:35.778764 929 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:35.778870 929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:35.779187 929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:35.781999 929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "a0b681d61c3c48a0b90c116bfd9dc3e8"
format_stamp: "Formatted at 2026-09-29 20:46:35 on dist-test-slave-blv7"
server_key: "1c5925325c3da93769f9dee07bdd5631"
server_key_iv: "f156fbe6b059adc0028dfa2911fcc377"
server_key_version: "encryptionkey@0"
I20260929 20:46:35.782707 929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "a0b681d61c3c48a0b90c116bfd9dc3e8"
format_stamp: "Formatted at 2026-09-29 20:46:35 on dist-test-slave-blv7"
server_key: "1c5925325c3da93769f9dee07bdd5631"
server_key_iv: "f156fbe6b059adc0028dfa2911fcc377"
server_key_version: "encryptionkey@0"
I20260929 20:46:35.787639 929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:46:35.790920 944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:35.792259 929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:46:35.792418 929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "a0b681d61c3c48a0b90c116bfd9dc3e8"
format_stamp: "Formatted at 2026-09-29 20:46:35 on dist-test-slave-blv7"
server_key: "1c5925325c3da93769f9dee07bdd5631"
server_key_iv: "f156fbe6b059adc0028dfa2911fcc377"
server_key_version: "encryptionkey@0"
I20260929 20:46:35.792588 929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:35.813354 929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:35.817123 929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:35.817376 929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:35.833235 929 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46257
I20260929 20:46:35.833253 995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46257 every 8 connection(s)
I20260929 20:46:35.834595 929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:46:35.838192 996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:35.840246 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 929
I20260929 20:46:35.840446 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:46:35.841059 388 external_mini_cluster.cc:1428] Setting key 36730f187617831d43d3f4ca51f77c1b
I20260929 20:46:35.850181 996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8: Bootstrap starting.
I20260929 20:46:35.853601 996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:35.854528 996 log.cc:824] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:46:35 dist-test-slave-blv7 krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:35.857352 996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8: No bootstrap required, opened a new log
I20260929 20:46:35.863116 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:35.842982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41582 (local address 127.0.97.62:46257)
0929 20:46:35.843715 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:35.843730 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:35.843778 (+ 48us) server_negotiation.cc:407] Connection header received
0929 20:46:35.845937 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:35.845964 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:35.846476 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:35.846898 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:35.848212 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:35.850043 (+ 1831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:35.851303 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:35.852301 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:35.852479 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:35.856298 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:35.856305 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:35.856319 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:35.856355 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:35.859269 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:35.860000 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:35.860006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:35.860012 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:35.860160 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:35.860514 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:35.860517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:35.860519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:35.861001 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:35.861199 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:35.861717 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 20:46:35.862100 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":259,"threads_started":1}
I20260929 20:46:35.865769 996 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b681d61c3c48a0b90c116bfd9dc3e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } }
I20260929 20:46:35.866142 996 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:35.866228 996 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0b681d61c3c48a0b90c116bfd9dc3e8, State: Initialized, Role: FOLLOWER
I20260929 20:46:35.866708 996 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b681d61c3c48a0b90c116bfd9dc3e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } }
I20260929 20:46:35.866899 996 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:46:35.866989 996 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:46:35.867137 996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:35.869364 996 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b681d61c3c48a0b90c116bfd9dc3e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } }
I20260929 20:46:35.869772 996 leader_election.cc:302] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0b681d61c3c48a0b90c116bfd9dc3e8; no voters:
I20260929 20:46:35.870262 996 leader_election.cc:288] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:46:35.870376 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:46:35.870967 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8 [term 1 LEADER]: Becoming Leader. State: Replica: a0b681d61c3c48a0b90c116bfd9dc3e8, State: Running, Role: LEADER
I20260929 20:46:35.871483 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b681d61c3c48a0b90c116bfd9dc3e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } }
I20260929 20:46:35.872041 996 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:46:35.874459 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0b681d61c3c48a0b90c116bfd9dc3e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b681d61c3c48a0b90c116bfd9dc3e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } } }
I20260929 20:46:35.874827 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8 [sys.catalog]: This master's current role is: LEADER
I20260929 20:46:35.875190 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0b681d61c3c48a0b90c116bfd9dc3e8. Latest consensus state: current_term: 1 leader_uuid: "a0b681d61c3c48a0b90c116bfd9dc3e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b681d61c3c48a0b90c116bfd9dc3e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46257 } } }
I20260929 20:46:35.875521 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8 [sys.catalog]: This master's current role is: LEADER
I20260929 20:46:35.876668 1008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:46:35.881680 1008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:46:35.889503 1008 catalog_manager.cc:1347] Generated new cluster ID: 88d2027d36c443bea0289484a3610dc3
I20260929 20:46:35.889598 1008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:46:35.916682 1008 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:46:35.918602 1008 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:46:35.925137 1008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0b681d61c3c48a0b90c116bfd9dc3e8: Generated new TSK 0
I20260929 20:46:35.926129 1008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:46:35.964416 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46257
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:36.266065 1022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:36.266340 1022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:36.266484 1022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:36.273485 1022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:36.273590 1022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:36.273728 1022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:46:36.283080 1022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46257
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:36.284313 1022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:36.285594 1022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:36.298062 1027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:36.297991 1030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:36.298017 1028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:36.298683 1022 server_base.cc:1034] running on GCE node
I20260929 20:46:36.299239 1022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:36.300051 1022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:36.301240 1022 hybrid_clock.cc:648] HybridClock initialized: now 1790714796301196 us; error 35 us; skew 500 ppm
I20260929 20:46:36.301530 1022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:36 dist-test-slave-blv7 krb5kdc[916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:36.304378 1022 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:46:36.305676 1022 webserver.cc:460] Webserver started at http://127.0.97.1:40555/ using document root <none> and password file <none>
I20260929 20:46:36.306304 1022 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:36.306407 1022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:36.306679 1022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:36.309438 1022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "c353f299fa244063b94256c8ede4c602"
format_stamp: "Formatted at 2026-09-29 20:46:36 on dist-test-slave-blv7"
server_key: "9eda9ef5c4ae5c917c0c059d29aa181a"
server_key_iv: "c6773ef20412b653db5036c8fffa8e5f"
server_key_version: "encryptionkey@0"
I20260929 20:46:36.310156 1022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c353f299fa244063b94256c8ede4c602"
format_stamp: "Formatted at 2026-09-29 20:46:36 on dist-test-slave-blv7"
server_key: "9eda9ef5c4ae5c917c0c059d29aa181a"
server_key_iv: "c6773ef20412b653db5036c8fffa8e5f"
server_key_version: "encryptionkey@0"
I20260929 20:46:36.314900 1022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260929 20:46:36.318033 1037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:36.319306 1022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 20:46:36.319447 1022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "c353f299fa244063b94256c8ede4c602"
format_stamp: "Formatted at 2026-09-29 20:46:36 on dist-test-slave-blv7"
server_key: "9eda9ef5c4ae5c917c0c059d29aa181a"
server_key_iv: "c6773ef20412b653db5036c8fffa8e5f"
server_key_version: "encryptionkey@0"
I20260929 20:46:36.319618 1022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:36.340718 1022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:36.344416 1022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:36.344657 1022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:36.345520 1022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:46:36.346729 1022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:46:36.346807 1022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:36.346923 1022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:46:36.346973 1022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:36.374972 1022 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34499
I20260929 20:46:36.375047 1149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34499 every 8 connection(s)
I20260929 20:46:36.376252 1022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:46:36.378072 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 1022
I20260929 20:46:36.378266 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:46:36.378976 388 external_mini_cluster.cc:1428] Setting key b4f0b4dfee8476bb56262fb703803230
Sep 29 20:46:36 dist-test-slave-blv7 krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 20:46:36.397392 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:36.378684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33659 (local address 127.0.97.62:46257)
0929 20:46:36.379128 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:36.379134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:36.380196 (+ 1062us) server_negotiation.cc:407] Connection header received
0929 20:46:36.381957 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:36.381963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:36.382040 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:36.382148 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:36.383656 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:36.384627 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:36.386787 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:36.387235 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:36.387326 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:36.391158 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:36.391165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:36.391170 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:36.391209 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:36.394252 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:36.395006 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:36.395012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:36.395042 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:36.395168 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:36.395726 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:36.395731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:36.395734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:36.396026 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:36.396135 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:36.396898 (+ 763us) server_negotiation.cc:299] Negotiation successful
0929 20:46:36.397087 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":207,"threads_started":1}
I20260929 20:46:36.398443 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:36.379295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46257 (local address 127.0.97.1:33659)
0929 20:46:36.380037 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:36.380082 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:36.381709 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:36.382338 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:36.382346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:36.382830 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:36.383501 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:36.383519 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:36.384780 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:36.384785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:36.386672 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:36.386687 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:36.387438 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:36.387444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:36.387705 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:36.388351 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:36.388376 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:36.390927 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:36.394377 (+ 3450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:36.394384 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:36.394397 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:36.394868 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:36.395296 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:36.395320 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:36.395324 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:36.395574 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:36.396148 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:36.396154 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:36.396513 (+ 359us) client_negotiation.cc:769] Sending connection context
0929 20:46:36.396814 (+ 301us) client_negotiation.cc:240] Negotiation successful
0929 20:46:36.397317 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":289,"threads_started":1}
I20260929 20:46:36.399866 1150 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46257
I20260929 20:46:36.400180 1150 heartbeater.cc:464] Registering TS with master...
I20260929 20:46:36.401006 1150 heartbeater.cc:511] Master 127.0.97.62:46257 requested a full tablet report, sending...
I20260929 20:46:36.403220 961 ts_manager.cc:194] Registered new tserver with Master: c353f299fa244063b94256c8ede4c602 (127.0.97.1:34499)
I20260929 20:46:36.405100 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:46:36.418092 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46257
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:36.721529 1156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:36.721807 1156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:36.721942 1156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:36.729120 1156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:36.729241 1156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:36.729390 1156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:46:36.739127 1156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46257
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:36.740404 1156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:36.741755 1156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:36.754381 1161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:36.754449 1164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:36.754420 1162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:36.755510 1156 server_base.cc:1034] running on GCE node
I20260929 20:46:36.756102 1156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:36.757055 1156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:36.758253 1156 hybrid_clock.cc:648] HybridClock initialized: now 1790714796758203 us; error 44 us; skew 500 ppm
I20260929 20:46:36.758554 1156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:36 dist-test-slave-blv7 krb5kdc[916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:36.761646 1156 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:46:36.763161 1156 webserver.cc:460] Webserver started at http://127.0.97.2:36315/ using document root <none> and password file <none>
I20260929 20:46:36.763820 1156 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:36.763931 1156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:36.764220 1156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:36.767096 1156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "9930639ebb0740fe855ec139f93881a3"
format_stamp: "Formatted at 2026-09-29 20:46:36 on dist-test-slave-blv7"
server_key: "1ee4dd91c601adb95adeeb26a8df0a40"
server_key_iv: "50c31425ea8d5cc433db77f741b1213a"
server_key_version: "encryptionkey@0"
I20260929 20:46:36.767872 1156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9930639ebb0740fe855ec139f93881a3"
format_stamp: "Formatted at 2026-09-29 20:46:36 on dist-test-slave-blv7"
server_key: "1ee4dd91c601adb95adeeb26a8df0a40"
server_key_iv: "50c31425ea8d5cc433db77f741b1213a"
server_key_version: "encryptionkey@0"
I20260929 20:46:36.772930 1156 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:46:36.776183 1171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:36.777576 1156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:46:36.777710 1156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "9930639ebb0740fe855ec139f93881a3"
format_stamp: "Formatted at 2026-09-29 20:46:36 on dist-test-slave-blv7"
server_key: "1ee4dd91c601adb95adeeb26a8df0a40"
server_key_iv: "50c31425ea8d5cc433db77f741b1213a"
server_key_version: "encryptionkey@0"
I20260929 20:46:36.777881 1156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:36.796309 1156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:36.800235 1156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:36.800482 1156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:36.801370 1156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:46:36.802666 1156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:46:36.802733 1156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:36.802841 1156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:46:36.802888 1156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:36.831394 1156 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37897
I20260929 20:46:36.831457 1283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37897 every 8 connection(s)
I20260929 20:46:36.832633 1156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:46:36.842720 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 1156
I20260929 20:46:36.842900 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:46:36.843391 388 external_mini_cluster.cc:1428] Setting key 34cef7bbec2b879370f4c10c82f5206a
Sep 29 20:46:36 dist-test-slave-blv7 krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:36.852669 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:36.835185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47825 (local address 127.0.97.62:46257)
0929 20:46:36.835336 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:36.835340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:36.836601 (+ 1261us) server_negotiation.cc:407] Connection header received
0929 20:46:36.838312 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:36.838317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:36.838421 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:36.838504 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:36.839921 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:36.840651 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:36.842100 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:36.842518 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:36.842619 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:36.846432 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:36.846438 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:36.846444 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:36.846480 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:36.849572 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:36.850388 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:36.850392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:36.850394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:36.850538 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:36.850987 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:36.850997 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:36.851000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:36.851349 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:36.851515 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:36.852153 (+ 638us) server_negotiation.cc:299] Negotiation successful
0929 20:46:36.852359 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 20:46:36.853668 1286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:36.835737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46257 (local address 127.0.97.2:47825)
0929 20:46:36.836454 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:36.836490 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:36.838064 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:36.838742 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:36.838751 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:36.839251 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:36.839770 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:36.839785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:36.840789 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:36.840793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:36.841971 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:36.841982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:36.842722 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:36.842729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:36.842960 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:36.843627 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:36.843653 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:36.846183 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:36.849707 (+ 3524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:36.849716 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:36.849734 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:36.850282 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:36.850634 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:36.850639 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:36.850643 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:36.850868 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:36.851487 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:36.851492 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:36.851782 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 20:46:36.852043 (+ 261us) client_negotiation.cc:240] Negotiation successful
0929 20:46:36.852541 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":293,"threads_started":1}
I20260929 20:46:36.854972 1284 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46257
I20260929 20:46:36.855264 1284 heartbeater.cc:464] Registering TS with master...
I20260929 20:46:36.855916 1284 heartbeater.cc:511] Master 127.0.97.62:46257 requested a full tablet report, sending...
I20260929 20:46:36.857265 961 ts_manager.cc:194] Registered new tserver with Master: 9930639ebb0740fe855ec139f93881a3 (127.0.97.2:37897)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:46:36.858367 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:46:36.879757 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46257
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:37.183815 1289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:37.184093 1289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:37.184201 1289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:37.191435 1289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:37.191540 1289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:37.191711 1289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:46:37.201483 1289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46257
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:37.202756 1289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:37.204155 1289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:37.216737 1295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:37.216699 1297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:37.216758 1294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:37.217368 1289 server_base.cc:1034] running on GCE node
I20260929 20:46:37.217962 1289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:37.218849 1289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:37.220036 1289 hybrid_clock.cc:648] HybridClock initialized: now 1790714797219971 us; error 56 us; skew 500 ppm
I20260929 20:46:37.220327 1289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:37 dist-test-slave-blv7 krb5kdc[916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:37.223361 1289 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:46:37.224686 1289 webserver.cc:460] Webserver started at http://127.0.97.3:33537/ using document root <none> and password file <none>
I20260929 20:46:37.225276 1289 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:37.225368 1289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:37.225611 1289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:37.228335 1289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "e935ff966ccc4e26b95d78038ac95808"
format_stamp: "Formatted at 2026-09-29 20:46:37 on dist-test-slave-blv7"
server_key: "c3be90121ff8add14d0875d0ad7467ce"
server_key_iv: "0e322d108a915a5ed28f6205e452990d"
server_key_version: "encryptionkey@0"
I20260929 20:46:37.229044 1289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e935ff966ccc4e26b95d78038ac95808"
format_stamp: "Formatted at 2026-09-29 20:46:37 on dist-test-slave-blv7"
server_key: "c3be90121ff8add14d0875d0ad7467ce"
server_key_iv: "0e322d108a915a5ed28f6205e452990d"
server_key_version: "encryptionkey@0"
I20260929 20:46:37.233735 1289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 20:46:37.236897 1304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:37.238142 1289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:46:37.238267 1289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "e935ff966ccc4e26b95d78038ac95808"
format_stamp: "Formatted at 2026-09-29 20:46:37 on dist-test-slave-blv7"
server_key: "c3be90121ff8add14d0875d0ad7467ce"
server_key_iv: "0e322d108a915a5ed28f6205e452990d"
server_key_version: "encryptionkey@0"
I20260929 20:46:37.238481 1289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:37.256319 1289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:37.260219 1289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:37.260469 1289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:37.261337 1289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:46:37.262575 1289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:46:37.262648 1289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:37.262759 1289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:46:37.262807 1289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:37.290774 1289 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43887
I20260929 20:46:37.290815 1416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43887 every 8 connection(s)
I20260929 20:46:37.291965 1289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:46:37.293356 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 1289
I20260929 20:46:37.293527 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:46:37.293915 388 external_mini_cluster.cc:1428] Setting key e994ba3835d287fb67225ffa875e4de4
Sep 29 20:46:37 dist-test-slave-blv7 krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:37.311676 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:37.294193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52513 (local address 127.0.97.62:46257)
0929 20:46:37.294370 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:37.294375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:37.295804 (+ 1429us) server_negotiation.cc:407] Connection header received
0929 20:46:37.297754 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:37.297759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:37.297838 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:37.297921 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:37.299375 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.300136 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.301640 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.302097 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.302228 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:37.305521 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:37.305527 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:37.305530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:37.305561 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:37.308707 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:37.309488 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:37.309492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:37.309494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:37.309601 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:37.310004 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:37.310008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:37.310010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:37.310268 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:37.310391 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:37.311123 (+ 732us) server_negotiation.cc:299] Negotiation successful
0929 20:46:37.311342 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:46:37.312618 1419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:37.294857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46257 (local address 127.0.97.3:52513)
0929 20:46:37.295667 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:37.295724 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:37.297520 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:37.298161 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:37.298168 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:37.298641 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:37.299228 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:37.299247 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.300282 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.300286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:37.301512 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:37.301523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.302323 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.302330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:37.302559 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:37.303210 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:37.303232 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:37.305294 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:37.308839 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:37.308849 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:37.308867 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:37.309367 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:37.309700 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:37.309705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:37.309709 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:37.309888 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:37.310384 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:37.310392 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:37.310731 (+ 339us) client_negotiation.cc:769] Sending connection context
0929 20:46:37.310987 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 20:46:37.311541 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":316,"threads_started":1}
I20260929 20:46:37.313908 1417 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46257
I20260929 20:46:37.314213 1417 heartbeater.cc:464] Registering TS with master...
I20260929 20:46:37.314965 1417 heartbeater.cc:511] Master 127.0.97.62:46257 requested a full tablet report, sending...
I20260929 20:46:37.316318 961 ts_manager.cc:194] Registered new tserver with Master: e935ff966ccc4e26b95d78038ac95808 (127.0.97.3:43887)
I20260929 20:46:37.317060 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52513
I20260929 20:46:37.321069 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:46:37.338228 1152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:37.326902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39670 (local address 127.0.97.62:46257)
0929 20:46:37.327086 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:37.327091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:37.327618 (+ 527us) server_negotiation.cc:407] Connection header received
0929 20:46:37.327755 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:37.327758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:37.327817 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:37.327883 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:37.329203 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.329895 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.331098 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.331534 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.331619 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:37.334047 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:37.334051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:37.334054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:37.334078 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:37.336259 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:37.336821 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:37.336825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:37.336827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:37.336931 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:37.337187 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:37.337189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:37.337191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:37.337460 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:37.337621 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:37.337869 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 20:46:37.338013 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:46:37.343891 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:46:37.350447 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:46:37.373320 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:37.362000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56236 (local address 127.0.97.1:34499)
0929 20:46:37.362501 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:37.362510 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:37.363776 (+ 1266us) server_negotiation.cc:407] Connection header received
0929 20:46:37.364173 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:37.364183 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:37.364451 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:37.364607 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:46:37.365517 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.367208 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.370006 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.371571 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.371720 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:37.371986 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:37.372715 (+ 729us) server_negotiation.cc:299] Negotiation successful
0929 20:46:37.372952 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":233,"threads_started":1}
I20260929 20:46:37.374496 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:37.361994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34499 (local address 127.0.0.1:56236)
0929 20:46:37.363613 (+ 1619us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:37.363662 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:37.363933 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:37.364946 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:37.364954 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:37.364985 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:46:37.365338 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:37.365348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.367374 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.367379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:37.369808 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:37.369822 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.372179 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.372186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:37.372412 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:37.372462 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 20:46:37.373944 (+ 1482us) client_negotiation.cc:240] Negotiation successful
0929 20:46:37.374170 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1369,"thread_start_us":1225,"threads_started":1}
I20260929 20:46:37.380872 1085 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2e23b90dca4323a1ef8ff684913570 (DEFAULT_TABLE table=test-table [id=04ca5bb29f9a46bdb8a85f5910c03435]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:46:37.382575 1085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2e23b90dca4323a1ef8ff684913570. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:37.382889 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:37.362394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37897 (local address 127.0.0.1:36962)
0929 20:46:37.375013 (+ 12619us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:37.375051 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:37.375160 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:37.376285 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:37.376290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:37.376313 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:46:37.376485 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:37.376495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.378502 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.378507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:37.380282 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:37.380293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.382119 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.382125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:37.382320 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:37.382349 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 20:46:37.382437 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 20:46:37.382570 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12493,"mutex_wait_us":16,"thread_start_us":14981,"threads_started":1}
I20260929 20:46:37.382970 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:37.362757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36962 (local address 127.0.97.2:37897)
0929 20:46:37.375655 (+ 12898us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:37.375665 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:37.375695 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:46:37.375795 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:37.375802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:37.376065 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:37.376654 (+ 589us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:46:37.376842 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.378342 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.380467 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.381914 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.382273 (+ 359us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:37.382464 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:37.382527 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 20:46:37.382667 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12761,"thread_start_us":12639,"threads_started":1}
I20260929 20:46:37.386699 1219 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2e23b90dca4323a1ef8ff684913570 (DEFAULT_TABLE table=test-table [id=04ca5bb29f9a46bdb8a85f5910c03435]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:46:37.388207 1219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2e23b90dca4323a1ef8ff684913570. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:37.389261 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:37.364572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42194 (local address 127.0.97.3:43887)
0929 20:46:37.376037 (+ 11465us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:37.376046 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:37.376622 (+ 576us) server_negotiation.cc:407] Connection header received
0929 20:46:37.376713 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:37.376721 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:37.376937 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:37.377112 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:46:37.378552 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.380124 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.383766 (+ 3642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.388252 (+ 4486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.388391 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:37.388638 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:37.388815 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 20:46:37.388967 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11303,"thread_start_us":11162,"threads_started":1}
I20260929 20:46:37.389297 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:37.364041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43887 (local address 127.0.0.1:42194)
0929 20:46:37.376392 (+ 12351us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:37.376408 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:37.376520 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:37.377361 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:37.377365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:37.377389 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:46:37.378354 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:37.378365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.380274 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.380279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:37.383585 (+ 3306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:37.383598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.388464 (+ 4866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.388470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:37.388642 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:37.388675 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:46:37.388763 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 20:46:37.388902 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12241,"mutex_wait_us":2,"thread_start_us":12119,"threads_started":1}
I20260929 20:46:37.392635 1352 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2e23b90dca4323a1ef8ff684913570 (DEFAULT_TABLE table=test-table [id=04ca5bb29f9a46bdb8a85f5910c03435]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:46:37.393764 1352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2e23b90dca4323a1ef8ff684913570. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:37.397584 1435 tablet_bootstrap.cc:492] T 8a2e23b90dca4323a1ef8ff684913570 P c353f299fa244063b94256c8ede4c602: Bootstrap starting.
I20260929 20:46:37.401105 1435 tablet_bootstrap.cc:654] T 8a2e23b90dca4323a1ef8ff684913570 P c353f299fa244063b94256c8ede4c602: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:37.401517 1436 tablet_bootstrap.cc:492] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3: Bootstrap starting.
I20260929 20:46:37.402403 1435 log.cc:824] T 8a2e23b90dca4323a1ef8ff684913570 P c353f299fa244063b94256c8ede4c602: Log is configured to *not* fsync() on all Append() calls
I20260929 20:46:37.404969 1436 tablet_bootstrap.cc:654] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:37.405539 1435 tablet_bootstrap.cc:492] T 8a2e23b90dca4323a1ef8ff684913570 P c353f299fa244063b94256c8ede4c602: No bootstrap required, opened a new log
I20260929 20:46:37.405809 1435 ts_tablet_manager.cc:1397] T 8a2e23b90dca4323a1ef8ff684913570 P c353f299fa244063b94256c8ede4c602: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260929 20:46:37.405896 1436 log.cc:824] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3: Log is configured to *not* fsync() on all Append() calls
I20260929 20:46:37.408140 1150 heartbeater.cc:503] Master 127.0.97.62:46257 was elected leader, sending a full tablet report...
I20260929 20:46:37.408488 1439 tablet_bootstrap.cc:492] T 8a2e23b90dca4323a1ef8ff684913570 P e935ff966ccc4e26b95d78038ac95808: Bootstrap starting.
I20260929 20:46:37.408630 1436 tablet_bootstrap.cc:492] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3: No bootstrap required, opened a new log
I20260929 20:46:37.408879 1436 ts_tablet_manager.cc:1397] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260929 20:46:37.411931 1439 tablet_bootstrap.cc:654] T 8a2e23b90dca4323a1ef8ff684913570 P e935ff966ccc4e26b95d78038ac95808: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:37.413086 1439 log.cc:824] T 8a2e23b90dca4323a1ef8ff684913570 P e935ff966ccc4e26b95d78038ac95808: Log is configured to *not* fsync() on all Append() calls
I20260929 20:46:37.415508 1435 raft_consensus.cc:348] T 8a2e23b90dca4323a1ef8ff684913570 P c353f299fa244063b94256c8ede4c602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c353f299fa244063b94256c8ede4c602" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34499 } } peers { permanent_uuid: "e935ff966ccc4e26b95d78038ac95808" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43887 } } peers { permanent_uuid: "9930639ebb0740fe855ec139f93881a3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37897 } }
I20260929 20:46:37.416065 1435 raft_consensus.cc:374] T 8a2e23b90dca4323a1ef8ff684913570 P c353f299fa244063b94256c8ede4c602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:37.416213 1435 raft_consensus.cc:729] T 8a2e23b90dca4323a1ef8ff684913570 P c353f299fa244063b94256c8ede4c602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c353f299fa244063b94256c8ede4c602, State: Initialized, Role: FOLLOWER
I20260929 20:46:37.417049 1439 tablet_bootstrap.cc:492] T 8a2e23b90dca4323a1ef8ff684913570 P e935ff966ccc4e26b95d78038ac95808: No bootstrap required, opened a new log
I20260929 20:46:37.417011 1435 consensus_queue.cc:260] T 8a2e23b90dca4323a1ef8ff684913570 P c353f299fa244063b94256c8ede4c602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c353f299fa244063b94256c8ede4c602" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34499 } } peers { permanent_uuid: "e935ff966ccc4e26b95d78038ac95808" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43887 } } peers { permanent_uuid: "9930639ebb0740fe855ec139f93881a3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37897 } }
I20260929 20:46:37.417351 1439 ts_tablet_manager.cc:1397] T 8a2e23b90dca4323a1ef8ff684913570 P e935ff966ccc4e26b95d78038ac95808: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 20:46:37.417764 1436 raft_consensus.cc:348] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c353f299fa244063b94256c8ede4c602" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34499 } } peers { permanent_uuid: "e935ff966ccc4e26b95d78038ac95808" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43887 } } peers { permanent_uuid: "9930639ebb0740fe855ec139f93881a3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37897 } }
I20260929 20:46:37.418234 1436 raft_consensus.cc:374] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:37.418362 1436 raft_consensus.cc:729] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9930639ebb0740fe855ec139f93881a3, State: Initialized, Role: FOLLOWER
I20260929 20:46:37.419090 1436 consensus_queue.cc:260] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c353f299fa244063b94256c8ede4c602" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34499 } } peers { permanent_uuid: "e935ff966ccc4e26b95d78038ac95808" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43887 } } peers { permanent_uuid: "9930639ebb0740fe855ec139f93881a3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37897 } }
I20260929 20:46:37.419700 1435 ts_tablet_manager.cc:1428] T 8a2e23b90dca4323a1ef8ff684913570 P c353f299fa244063b94256c8ede4c602: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260929 20:46:37.420593 1284 heartbeater.cc:503] Master 127.0.97.62:46257 was elected leader, sending a full tablet report...
I20260929 20:46:37.421161 1436 ts_tablet_manager.cc:1428] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260929 20:46:37.426226 1439 raft_consensus.cc:348] T 8a2e23b90dca4323a1ef8ff684913570 P e935ff966ccc4e26b95d78038ac95808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c353f299fa244063b94256c8ede4c602" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34499 } } peers { permanent_uuid: "e935ff966ccc4e26b95d78038ac95808" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43887 } } peers { permanent_uuid: "9930639ebb0740fe855ec139f93881a3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37897 } }
I20260929 20:46:37.426564 1439 raft_consensus.cc:374] T 8a2e23b90dca4323a1ef8ff684913570 P e935ff966ccc4e26b95d78038ac95808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:37.426651 1439 raft_consensus.cc:729] T 8a2e23b90dca4323a1ef8ff684913570 P e935ff966ccc4e26b95d78038ac95808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e935ff966ccc4e26b95d78038ac95808, State: Initialized, Role: FOLLOWER
I20260929 20:46:37.427186 1439 consensus_queue.cc:260] T 8a2e23b90dca4323a1ef8ff684913570 P e935ff966ccc4e26b95d78038ac95808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c353f299fa244063b94256c8ede4c602" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34499 } } peers { permanent_uuid: "e935ff966ccc4e26b95d78038ac95808" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43887 } } peers { permanent_uuid: "9930639ebb0740fe855ec139f93881a3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37897 } }
I20260929 20:46:37.428434 1417 heartbeater.cc:503] Master 127.0.97.62:46257 was elected leader, sending a full tablet report...
I20260929 20:46:37.428838 1439 ts_tablet_manager.cc:1428] T 8a2e23b90dca4323a1ef8ff684913570 P e935ff966ccc4e26b95d78038ac95808: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260929 20:46:37.490492 1442 raft_consensus.cc:482] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:46:37.490736 1442 raft_consensus.cc:504] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c353f299fa244063b94256c8ede4c602" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34499 } } peers { permanent_uuid: "e935ff966ccc4e26b95d78038ac95808" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43887 } } peers { permanent_uuid: "9930639ebb0740fe855ec139f93881a3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37897 } }
I20260929 20:46:37.492261 1442 leader_election.cc:288] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c353f299fa244063b94256c8ede4c602 (127.0.97.1:34499), e935ff966ccc4e26b95d78038ac95808 (127.0.97.3:43887)
I20260929 20:46:37.499630 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:37.492614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35813 (local address 127.0.97.1:34499)
0929 20:46:37.492771 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:37.492775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:37.493426 (+ 651us) server_negotiation.cc:407] Connection header received
0929 20:46:37.493649 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:37.493654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:37.493727 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:37.493811 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:46:37.494457 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.495263 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.497424 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.498386 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.498511 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:37.498600 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:37.499269 (+ 669us) server_negotiation.cc:299] Negotiation successful
0929 20:46:37.499392 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 20:46:37.499661 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:37.492581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34499 (local address 127.0.97.2:35813)
0929 20:46:37.493332 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:37.493358 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:37.493489 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:37.493924 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:37.493929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:37.493953 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:46:37.494295 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:37.494304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.495385 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.495389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:37.497301 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:37.497315 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.498984 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.498990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:37.499131 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:37.499155 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:46:37.499226 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 20:46:37.499318 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":28,"thread_start_us":413,"threads_started":1}
I20260929 20:46:37.500242 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:37.493679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49521 (local address 127.0.97.3:43887)
0929 20:46:37.493857 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:37.493862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:37.494092 (+ 230us) server_negotiation.cc:407] Connection header received
0929 20:46:37.494230 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:37.494234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:37.494286 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:37.494366 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:46:37.495085 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.495886 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.497901 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.499379 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.499504 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:37.499593 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:37.499945 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 20:46:37.500059 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:46:37.500298 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:37.493450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43887 (local address 127.0.97.2:49521)
0929 20:46:37.494001 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:37.494019 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:37.494133 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:37.494660 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:37.494663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:37.494681 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:46:37.494955 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:37.494962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.496034 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.496039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:37.497761 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:37.497770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.499577 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.499583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:37.499762 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:37.499804 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 20:46:37.499880 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 20:46:37.499987 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":1,"thread_start_us":269,"threads_started":1}
I20260929 20:46:37.500646 1105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2e23b90dca4323a1ef8ff684913570" candidate_uuid: "9930639ebb0740fe855ec139f93881a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c353f299fa244063b94256c8ede4c602" is_pre_election: true
I20260929 20:46:37.501166 1105 raft_consensus.cc:2454] T 8a2e23b90dca4323a1ef8ff684913570 P c353f299fa244063b94256c8ede4c602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9930639ebb0740fe855ec139f93881a3 in term 0.
I20260929 20:46:37.501257 1372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2e23b90dca4323a1ef8ff684913570" candidate_uuid: "9930639ebb0740fe855ec139f93881a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e935ff966ccc4e26b95d78038ac95808" is_pre_election: true
I20260929 20:46:37.501709 1372 raft_consensus.cc:2454] T 8a2e23b90dca4323a1ef8ff684913570 P e935ff966ccc4e26b95d78038ac95808 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9930639ebb0740fe855ec139f93881a3 in term 0.
I20260929 20:46:37.501818 1173 leader_election.cc:302] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9930639ebb0740fe855ec139f93881a3, c353f299fa244063b94256c8ede4c602; no voters:
I20260929 20:46:37.502193 1442 raft_consensus.cc:2781] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:46:37.502314 1442 raft_consensus.cc:482] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:46:37.502410 1442 raft_consensus.cc:3037] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:37.504470 1442 raft_consensus.cc:504] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c353f299fa244063b94256c8ede4c602" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34499 } } peers { permanent_uuid: "e935ff966ccc4e26b95d78038ac95808" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43887 } } peers { permanent_uuid: "9930639ebb0740fe855ec139f93881a3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37897 } }
I20260929 20:46:37.504981 1442 leader_election.cc:288] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3 [CANDIDATE]: Term 1 election: Requested vote from peers c353f299fa244063b94256c8ede4c602 (127.0.97.1:34499), e935ff966ccc4e26b95d78038ac95808 (127.0.97.3:43887)
I20260929 20:46:37.505409 1105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2e23b90dca4323a1ef8ff684913570" candidate_uuid: "9930639ebb0740fe855ec139f93881a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c353f299fa244063b94256c8ede4c602"
I20260929 20:46:37.505465 1372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2e23b90dca4323a1ef8ff684913570" candidate_uuid: "9930639ebb0740fe855ec139f93881a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e935ff966ccc4e26b95d78038ac95808"
I20260929 20:46:37.505581 1105 raft_consensus.cc:3037] T 8a2e23b90dca4323a1ef8ff684913570 P c353f299fa244063b94256c8ede4c602 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:37.505612 1372 raft_consensus.cc:3037] T 8a2e23b90dca4323a1ef8ff684913570 P e935ff966ccc4e26b95d78038ac95808 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:37.507663 1372 raft_consensus.cc:2454] T 8a2e23b90dca4323a1ef8ff684913570 P e935ff966ccc4e26b95d78038ac95808 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9930639ebb0740fe855ec139f93881a3 in term 1.
I20260929 20:46:37.507689 1105 raft_consensus.cc:2454] T 8a2e23b90dca4323a1ef8ff684913570 P c353f299fa244063b94256c8ede4c602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9930639ebb0740fe855ec139f93881a3 in term 1.
I20260929 20:46:37.508008 1173 leader_election.cc:302] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9930639ebb0740fe855ec139f93881a3, e935ff966ccc4e26b95d78038ac95808; no voters:
I20260929 20:46:37.508244 1442 raft_consensus.cc:2781] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:46:37.508651 1442 raft_consensus.cc:686] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3 [term 1 LEADER]: Becoming Leader. State: Replica: 9930639ebb0740fe855ec139f93881a3, State: Running, Role: LEADER
I20260929 20:46:37.509142 1442 consensus_queue.cc:237] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c353f299fa244063b94256c8ede4c602" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34499 } } peers { permanent_uuid: "e935ff966ccc4e26b95d78038ac95808" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43887 } } peers { permanent_uuid: "9930639ebb0740fe855ec139f93881a3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37897 } }
I20260929 20:46:37.513136 961 catalog_manager.cc:5495] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9930639ebb0740fe855ec139f93881a3 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "9930639ebb0740fe855ec139f93881a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c353f299fa244063b94256c8ede4c602" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e935ff966ccc4e26b95d78038ac95808" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9930639ebb0740fe855ec139f93881a3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37897 } health_report { overall_health: HEALTHY } } }
Sep 29 20:46:37 dist-test-slave-blv7 krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:46:37.535116 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:37.521089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56244 (local address 127.0.97.1:34499)
0929 20:46:37.521223 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:37.521227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:37.521695 (+ 468us) server_negotiation.cc:407] Connection header received
0929 20:46:37.521851 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:37.521855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:37.521930 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:37.522008 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:37.523566 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.524385 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.525569 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.525985 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.526088 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:37.529660 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:37.529679 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:37.529695 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:37.529730 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:37.532618 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:37.533212 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:37.533220 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:37.533222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:37.533339 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:37.533676 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:37.533680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:37.533682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:37.534039 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:37.534297 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:37.534575 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 20:46:37.534772 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 20:46:37.544158 1418 tablet.cc:2369] T 8a2e23b90dca4323a1ef8ff684913570 P e935ff966ccc4e26b95d78038ac95808: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 20:46:37 dist-test-slave-blv7 krb5kdc[916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714797, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:46:37 dist-test-slave-blv7 krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714797, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:46:37.575315 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:37.562331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56246 (local address 127.0.97.1:34499)
0929 20:46:37.562552 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:37.562558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:37.562705 (+ 147us) server_negotiation.cc:407] Connection header received
0929 20:46:37.562847 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:37.562851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:37.562934 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:37.563040 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:37.564263 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.564975 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.566171 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.566585 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.566688 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:37.570663 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:37.570668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:37.570671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:37.570706 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:37.573171 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:37.573750 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:37.573753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:37.573756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:37.573872 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:37.574171 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:37.574173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:37.574175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:37.574419 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:37.574615 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:37.574830 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 20:46:37.574985 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260929 20:46:37.584574 1285 tablet.cc:2369] T 8a2e23b90dca4323a1ef8ff684913570 P 9930639ebb0740fe855ec139f93881a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 20:46:37 dist-test-slave-blv7 krb5kdc[916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:46:37 dist-test-slave-blv7 krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:46:37.615361 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:37.602646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56248 (local address 127.0.97.1:34499)
0929 20:46:37.602826 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:37.602833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:37.603197 (+ 364us) server_negotiation.cc:407] Connection header received
0929 20:46:37.603367 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:37.603371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:37.603432 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:37.603512 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:37.604768 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.605448 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.606617 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.607071 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.607174 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:37.610662 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:37.610667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:37.610670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:37.610701 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:37.613026 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:37.613593 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:37.613597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:37.613599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:37.613710 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:37.614026 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:37.614031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:37.614034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:37.614305 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:37.614528 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:37.614790 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 20:46:37.615007 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 20:46:37.617017 1065 tablet_service.cc:2908] Rejecting scan request for tablet 8a2e23b90dca4323a1ef8ff684913570: Uninitialized: clean time has not yet been initialized
I20260929 20:46:37.619598 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 1022
W20260929 20:46:37.628279 1151 tablet.cc:2369] T 8a2e23b90dca4323a1ef8ff684913570 P c353f299fa244063b94256c8ede4c602: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:46:37.630743 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:37.620488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56260 (local address 127.0.97.1:34499)
0929 20:46:37.620628 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:37.620631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:37.620968 (+ 337us) server_negotiation.cc:407] Connection header received
0929 20:46:37.621093 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:37.621097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:37.621156 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:37.621232 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:37.622240 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.622899 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.624085 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.624483 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.624583 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:37.626586 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:37.626590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:37.626593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:37.626621 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:37.628870 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:37.629395 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:37.629399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:37.629401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:37.629505 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:37.629759 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:37.629762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:37.629763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:37.630031 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:37.630222 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:37.630384 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 20:46:37.630531 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 20:46:37.631296 1145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:46:37.718600 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 1022
I20260929 20:46:37.731607 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 1156
Sep 29 20:46:37 dist-test-slave-blv7 krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:46:37.744959 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:37.732284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36970 (local address 127.0.97.2:37897)
0929 20:46:37.732482 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:37.732487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:37.732504 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:46:37.732593 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:37.732597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:37.732655 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:37.732739 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:37.733777 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.734488 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.735735 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.736115 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.736201 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:37.739854 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:37.739868 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:37.739884 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:37.739940 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:37.742568 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:37.743197 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:37.743204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:37.743207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:37.743338 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:37.743689 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:37.743693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:37.743695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:37.744073 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:37.744304 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:37.744466 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 20:46:37.744649 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:46:37.745563 1279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:46:37.835676 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 1156
I20260929 20:46:37.854844 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 1289
Sep 29 20:46:37 dist-test-slave-blv7 krb5kdc[916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:46:37.867753 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:37.855537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42200 (local address 127.0.97.3:43887)
0929 20:46:37.855681 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:37.855685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:37.855759 (+ 74us) server_negotiation.cc:407] Connection header received
0929 20:46:37.855895 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:37.855900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:37.855970 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:37.856049 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:37.857146 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.857832 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.858985 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:37.859413 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:37.859499 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:37.863110 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:37.863123 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:37.863139 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:37.863170 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:37.865653 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:37.866189 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:37.866197 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:37.866199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:37.866324 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:37.866588 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:37.866591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:37.866593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:37.866916 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:37.867190 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:37.867347 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 20:46:37.867523 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 20:46:37.868343 1412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:46:37.952323 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 1289
I20260929 20:46:37.964628 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 929
I20260929 20:46:37.965627 991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:46:38.067675 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 929
2026-09-29T20:46:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (7820 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:46:38 dist-test-slave-blv7 krb5kdc[1465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:46:38 dist-test-slave-blv7 krb5kdc[1465](info): set up 2 sockets
Sep 29 20:46:38 dist-test-slave-blv7 krb5kdc[1465](info): commencing operation
krb5kdc: starting...
W20260929 20:46:39.058775 388 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:46:39 dist-test-slave-blv7 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:46:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:46:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:46:39.216620 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45269
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:39.511478 1478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:39.511766 1478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:39.511902 1478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:39.519001 1478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:46:39.519126 1478 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 20:46:39.519192 1478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:39.519276 1478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:39.519341 1478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:46:39.519418 1478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:46:39.529086 1478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45269
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:39.530290 1478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:39.531553 1478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:39.542256 1483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:39.542285 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:39.542330 1484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:39.543232 1478 server_base.cc:1034] running on GCE node
I20260929 20:46:39.543922 1478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:39.545125 1478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:39.546315 1478 hybrid_clock.cc:648] HybridClock initialized: now 1790714799546259 us; error 35 us; skew 500 ppm
I20260929 20:46:39.546602 1478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:39 dist-test-slave-blv7 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:39.553370 1478 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:46:39.554651 1478 webserver.cc:460] Webserver started at http://127.0.97.62:42461/ using document root <none> and password file <none>
I20260929 20:46:39.555338 1478 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:39.555428 1478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:39.555708 1478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:39.558456 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "054502fd0ff74352bbe80ef9027f0476"
format_stamp: "Formatted at 2026-09-29 20:46:39 on dist-test-slave-blv7"
server_key: "07039f66b4b0211a6312ab32aeca5ebc"
server_key_iv: "941eef6337354210a61c9606091bedc4"
server_key_version: "encryptionkey@0"
I20260929 20:46:39.559222 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "054502fd0ff74352bbe80ef9027f0476"
format_stamp: "Formatted at 2026-09-29 20:46:39 on dist-test-slave-blv7"
server_key: "07039f66b4b0211a6312ab32aeca5ebc"
server_key_iv: "941eef6337354210a61c9606091bedc4"
server_key_version: "encryptionkey@0"
I20260929 20:46:39.563716 1478 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:46:39.566728 1493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:39.567919 1478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 20:46:39.568063 1478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "054502fd0ff74352bbe80ef9027f0476"
format_stamp: "Formatted at 2026-09-29 20:46:39 on dist-test-slave-blv7"
server_key: "07039f66b4b0211a6312ab32aeca5ebc"
server_key_iv: "941eef6337354210a61c9606091bedc4"
server_key_version: "encryptionkey@0"
I20260929 20:46:39.568238 1478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:39.599138 1478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:39.602772 1478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:39.603008 1478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:39.622932 1478 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45269
I20260929 20:46:39.622908 1554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45269 every 8 connection(s)
I20260929 20:46:39.624344 1478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:46:39.627902 1555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:39.629706 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 1478
I20260929 20:46:39.629915 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:46:39.630553 388 external_mini_cluster.cc:1428] Setting key 2d29b54c9e9a0b304938811884e07496
I20260929 20:46:39.638693 1555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476: Bootstrap starting.
I20260929 20:46:39.642083 1555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:39.643283 1555 log.cc:824] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:46:39 dist-test-slave-blv7 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:39.646121 1555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476: No bootstrap required, opened a new log
I20260929 20:46:39.652426 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:39.632485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47504 (local address 127.0.97.62:45269)
0929 20:46:39.633286 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:39.633297 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:39.633339 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:46:39.635101 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:39.635128 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:39.635529 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:39.635881 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:39.637417 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:39.638668 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:39.640357 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:39.641018 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:39.641146 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:39.645684 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:39.645691 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:39.645705 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:39.645737 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:39.648667 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:39.649390 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:39.649396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:39.649401 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:39.649564 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:39.649910 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:39.649914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:39.649916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:39.650410 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:39.650642 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:39.651010 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 20:46:39.651478 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":376,"threads_started":1}
I20260929 20:46:39.654125 1555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054502fd0ff74352bbe80ef9027f0476" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45269 } }
I20260929 20:46:39.654448 1555 raft_consensus.cc:374] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:39.654537 1555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 054502fd0ff74352bbe80ef9027f0476, State: Initialized, Role: FOLLOWER
I20260929 20:46:39.655125 1555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054502fd0ff74352bbe80ef9027f0476" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45269 } }
I20260929 20:46:39.655332 1555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:46:39.655458 1555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:46:39.655596 1555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:39.657581 1555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054502fd0ff74352bbe80ef9027f0476" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45269 } }
I20260929 20:46:39.658001 1555 leader_election.cc:302] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 054502fd0ff74352bbe80ef9027f0476; no voters:
I20260929 20:46:39.658485 1555 leader_election.cc:288] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:46:39.658592 1560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:46:39.659507 1560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476 [term 1 LEADER]: Becoming Leader. State: Replica: 054502fd0ff74352bbe80ef9027f0476, State: Running, Role: LEADER
I20260929 20:46:39.660112 1560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054502fd0ff74352bbe80ef9027f0476" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45269 } }
I20260929 20:46:39.660559 1555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:46:39.662959 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 054502fd0ff74352bbe80ef9027f0476. Latest consensus state: current_term: 1 leader_uuid: "054502fd0ff74352bbe80ef9027f0476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054502fd0ff74352bbe80ef9027f0476" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45269 } } }
I20260929 20:46:39.663424 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476 [sys.catalog]: This master's current role is: LEADER
I20260929 20:46:39.662928 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "054502fd0ff74352bbe80ef9027f0476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054502fd0ff74352bbe80ef9027f0476" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45269 } } }
I20260929 20:46:39.664294 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476 [sys.catalog]: This master's current role is: LEADER
I20260929 20:46:39.668536 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:46:39.673367 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:46:39.681888 1568 catalog_manager.cc:1347] Generated new cluster ID: af9cd291a1c64f50a319d62243dd84a4
I20260929 20:46:39.681993 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:46:39.699411 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:46:39.701143 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:46:39.708436 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476: Generated new TSK 0
I20260929 20:46:39.709390 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:46:39.748309 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45269
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:40.048228 1581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:40.048485 1581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:40.048621 1581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:40.055537 1581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:40.055644 1581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:40.055759 1581 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:46:40.055874 1581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:46:40.065325 1581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:40.066558 1581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:40.067824 1581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:40.079989 1589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:40.079937 1587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:40.079954 1586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:40.080603 1581 server_base.cc:1034] running on GCE node
I20260929 20:46:40.081125 1581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:40.081943 1581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:40.083120 1581 hybrid_clock.cc:648] HybridClock initialized: now 1790714800083061 us; error 33 us; skew 500 ppm
I20260929 20:46:40.083396 1581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:40 dist-test-slave-blv7 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:40.086216 1581 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:46:40.087471 1581 webserver.cc:460] Webserver started at http://127.0.97.1:34885/ using document root <none> and password file <none>
I20260929 20:46:40.088070 1581 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:40.088161 1581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:40.088452 1581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:40.091089 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "de2c783b8c4c401ba731af02a500a22a"
format_stamp: "Formatted at 2026-09-29 20:46:40 on dist-test-slave-blv7"
server_key: "6c48d4279be63e6aac8ed967c2999533"
server_key_iv: "275b13016e624fd7068434d164645211"
server_key_version: "encryptionkey@0"
I20260929 20:46:40.091809 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "de2c783b8c4c401ba731af02a500a22a"
format_stamp: "Formatted at 2026-09-29 20:46:40 on dist-test-slave-blv7"
server_key: "6c48d4279be63e6aac8ed967c2999533"
server_key_iv: "275b13016e624fd7068434d164645211"
server_key_version: "encryptionkey@0"
I20260929 20:46:40.096422 1581 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:46:40.099501 1596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:40.100693 1581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 20:46:40.100831 1581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "de2c783b8c4c401ba731af02a500a22a"
format_stamp: "Formatted at 2026-09-29 20:46:40 on dist-test-slave-blv7"
server_key: "6c48d4279be63e6aac8ed967c2999533"
server_key_iv: "275b13016e624fd7068434d164645211"
server_key_version: "encryptionkey@0"
I20260929 20:46:40.101015 1581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:40.121551 1581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:40.125149 1581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:40.125381 1581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:40.127892 1581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:46:40.127985 1581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:40.128106 1581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:46:40.128176 1581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:46:40 dist-test-slave-blv7 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:40.147712 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:40.128458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48731 (local address 127.0.97.62:45269)
0929 20:46:40.128683 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:40.128689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:40.129959 (+ 1270us) server_negotiation.cc:407] Connection header received
0929 20:46:40.131945 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:40.131951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:40.132026 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:40.132108 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:40.134084 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:40.134973 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:40.137361 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:40.137776 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:40.137900 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:40.142258 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:40.142265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:40.142270 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:40.142309 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:40.145016 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:40.145644 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:40.145649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:40.145653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:40.145771 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:40.146145 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:40.146149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:40.146153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:40.146412 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:40.146537 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:40.147189 (+ 652us) server_negotiation.cc:299] Negotiation successful
0929 20:46:40.147396 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 20:46:40.148706 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:40.129033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45269 (local address 127.0.97.1:48731)
0929 20:46:40.129797 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:40.129838 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:40.131696 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:40.132367 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:40.132379 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:40.133065 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:40.133905 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:40.133938 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:40.135176 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:40.135180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:40.137233 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:40.137244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:40.138053 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:40.138065 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:40.138335 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:40.139101 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:40.139140 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:40.142021 (+ 2881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:40.145135 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:40.145144 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:40.145161 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:40.145540 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:40.145864 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:40.145868 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:40.145872 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:40.146050 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:40.146527 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:40.146534 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:40.146845 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 20:46:40.147124 (+ 279us) client_negotiation.cc:240] Negotiation successful
0929 20:46:40.147633 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":346,"threads_started":1}
I20260929 20:46:40.167436 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:40.155068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45475 (local address 127.0.97.62:45269)
0929 20:46:40.155628 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:40.155633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:40.155808 (+ 175us) server_negotiation.cc:407] Connection header received
0929 20:46:40.155993 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:40.155998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:40.156071 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:40.156173 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:40.157528 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:40.158376 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:40.160360 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:40.160759 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:40.160892 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:40.162507 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:40.162511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:40.162514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:40.162541 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:40.165095 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:40.165660 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:40.165665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:40.165667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:40.165778 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:40.166105 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:40.166110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:40.166113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:40.166358 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:40.166495 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:40.166830 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 20:46:40.167049 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452}
I20260929 20:46:40.167547 1666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:40.155051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45269 (local address 127.0.97.1:45475)
0929 20:46:40.155696 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:40.155716 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:40.155871 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:40.156211 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:40.156216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:40.156934 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:40.157367 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:40.157380 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:40.158509 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:40.158514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:40.160248 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:40.160261 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:40.160943 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:40.160950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:40.161098 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:40.161624 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:40.161639 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:40.162346 (+ 707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:40.165206 (+ 2860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:40.165211 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:40.165215 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:40.165561 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:40.165871 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:40.165876 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:40.165880 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:40.166014 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:40.166460 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:40.166464 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:40.166625 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 20:46:40.166736 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 20:46:40.167054 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":342,"threads_started":1}
I20260929 20:46:40.178622 1581 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41333
I20260929 20:46:40.178648 1715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41333 every 8 connection(s)
I20260929 20:46:40.179769 1581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:46:40.181504 1716 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45269
I20260929 20:46:40.181675 1716 heartbeater.cc:464] Registering TS with master...
I20260929 20:46:40.182287 1716 heartbeater.cc:511] Master 127.0.97.62:45269 requested a full tablet report, sending...
I20260929 20:46:40.183602 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 1581
I20260929 20:46:40.183789 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:46:40.184079 1510 ts_manager.cc:194] Registered new tserver with Master: de2c783b8c4c401ba731af02a500a22a (127.0.97.1:41333)
I20260929 20:46:40.184369 388 external_mini_cluster.cc:1428] Setting key 4662fe0db1cc144086a4f34de8b3bf19
I20260929 20:46:40.185734 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48731
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:46:40.218240 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45269
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:40.516813 1720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:40.517081 1720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:40.517222 1720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:40.524255 1720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:40.524353 1720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:40.524473 1720 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:46:40.524569 1720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:46:40.533846 1720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:40.535077 1720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:40.536345 1720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:40.548250 1725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:40.548223 1726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:40.548272 1728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:40.549167 1720 server_base.cc:1034] running on GCE node
I20260929 20:46:40.549691 1720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:40.550562 1720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:40.551753 1720 hybrid_clock.cc:648] HybridClock initialized: now 1790714800551676 us; error 61 us; skew 500 ppm
I20260929 20:46:40.552023 1720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:40 dist-test-slave-blv7 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:40.554841 1720 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:46:40.556208 1720 webserver.cc:460] Webserver started at http://127.0.97.2:33319/ using document root <none> and password file <none>
I20260929 20:46:40.556828 1720 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:40.556928 1720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:40.557205 1720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:40.559979 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "63969b25b0244237b7729b899aea297c"
format_stamp: "Formatted at 2026-09-29 20:46:40 on dist-test-slave-blv7"
server_key: "0ed1d8b3081e6a4a401470a87cb5a232"
server_key_iv: "06b98c2bda3cc568b9dde38b429a2e5e"
server_key_version: "encryptionkey@0"
I20260929 20:46:40.560750 1720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "63969b25b0244237b7729b899aea297c"
format_stamp: "Formatted at 2026-09-29 20:46:40 on dist-test-slave-blv7"
server_key: "0ed1d8b3081e6a4a401470a87cb5a232"
server_key_iv: "06b98c2bda3cc568b9dde38b429a2e5e"
server_key_version: "encryptionkey@0"
I20260929 20:46:40.565503 1720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:46:40.568759 1735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:40.569969 1720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:46:40.570098 1720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "63969b25b0244237b7729b899aea297c"
format_stamp: "Formatted at 2026-09-29 20:46:40 on dist-test-slave-blv7"
server_key: "0ed1d8b3081e6a4a401470a87cb5a232"
server_key_iv: "06b98c2bda3cc568b9dde38b429a2e5e"
server_key_version: "encryptionkey@0"
I20260929 20:46:40.570271 1720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:40.584954 1720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:40.588668 1720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:40.588888 1720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:40.591354 1720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:46:40.591449 1720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:40.591579 1720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:46:40.591686 1720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 29 20:46:40 dist-test-slave-blv7 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:40.611737 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:40.591722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52805 (local address 127.0.97.62:45269)
0929 20:46:40.591890 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:40.591896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:40.593282 (+ 1386us) server_negotiation.cc:407] Connection header received
0929 20:46:40.595246 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:40.595251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:40.595329 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:40.595413 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:40.597330 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:40.598258 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:40.600976 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:40.601470 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:40.601562 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:40.606328 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:40.606333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:40.606336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:40.606366 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:40.608959 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:40.609608 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:40.609613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:40.609617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:40.609733 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:40.610144 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:40.610149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:40.610153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:40.610405 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:40.610538 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:40.611193 (+ 655us) server_negotiation.cc:299] Negotiation successful
0929 20:46:40.611418 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:46:40.613237 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:40.592355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45269 (local address 127.0.97.2:52805)
0929 20:46:40.593127 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:40.593162 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:40.594942 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:40.595665 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:40.595673 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:40.596356 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:40.597162 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:40.597189 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:40.598441 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:40.598447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:40.600825 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:40.600840 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:40.601689 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:40.601697 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:40.601987 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:40.602725 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:40.602759 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:40.606108 (+ 3349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:40.609077 (+ 2969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:40.609088 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:40.609104 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:40.609493 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:40.609827 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:40.609832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:40.609836 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:40.610035 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:40.610515 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:40.610523 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:40.610851 (+ 328us) client_negotiation.cc:769] Sending connection context
0929 20:46:40.611094 (+ 243us) client_negotiation.cc:240] Negotiation successful
0929 20:46:40.611623 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":328,"threads_started":1}
I20260929 20:46:40.633620 1720 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44427
I20260929 20:46:40.633641 1854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44427 every 8 connection(s)
I20260929 20:46:40.635319 1720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:46:40.638897 1855 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45269
I20260929 20:46:40.639122 1855 heartbeater.cc:464] Registering TS with master...
I20260929 20:46:40.639858 1855 heartbeater.cc:511] Master 127.0.97.62:45269 requested a full tablet report, sending...
I20260929 20:46:40.641165 1510 ts_manager.cc:194] Registered new tserver with Master: 63969b25b0244237b7729b899aea297c (127.0.97.2:44427)
I20260929 20:46:40.641832 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:40.625979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37201 (local address 127.0.97.62:45269)
0929 20:46:40.626147 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:40.626153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:40.628099 (+ 1946us) server_negotiation.cc:407] Connection header received
0929 20:46:40.628282 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:40.628287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:40.628362 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:40.628461 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:40.629832 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:40.630926 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:40.632967 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:40.633621 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:40.633723 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:40.635873 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:40.635879 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:40.635884 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:40.635920 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:40.639203 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:40.639903 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:40.639909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:40.639913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:40.640060 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:40.640385 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:40.640395 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:40.640399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:40.640750 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:40.640937 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:40.641272 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 20:46:40.641464 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:46:40.641841 1836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:40.626488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45269 (local address 127.0.97.2:37201)
0929 20:46:40.627947 (+ 1459us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:40.627992 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:40.628161 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:40.628627 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:40.628632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:40.629249 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:40.629660 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:40.629672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:40.631103 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:40.631109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:40.632801 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:40.632815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:40.633767 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:40.633773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:40.633957 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:40.634569 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:40.634591 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:40.635680 (+ 1089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:40.639346 (+ 3666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:40.639368 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:40.639373 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:40.639782 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:40.640155 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:40.640160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:40.640164 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:40.640281 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:40.640857 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:40.640861 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:40.641039 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 20:46:40.641148 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 20:46:40.641445 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1292,"thread_start_us":1156,"threads_started":1}
I20260929 20:46:40.641891 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 1720
I20260929 20:46:40.642050 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:46:40.642153 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52805
I20260929 20:46:40.642618 388 external_mini_cluster.cc:1428] Setting key 24fbf299223440606a3e5a82569f8818
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:46:40.677614 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45269
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:40.976475 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:40.976743 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:40.976863 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:40.984061 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:40.984169 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:40.984265 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:46:40.984349 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:46:40.993762 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:40.994913 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:40.996244 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:41.008737 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:41.008935 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:41.008913 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:41.009912 1859 server_base.cc:1034] running on GCE node
I20260929 20:46:41.010468 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:41.011389 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:41.012609 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790714801012541 us; error 42 us; skew 500 ppm
I20260929 20:46:41.012876 1859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:41 dist-test-slave-blv7 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:41.015873 1859 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:46:41.017189 1859 webserver.cc:460] Webserver started at http://127.0.97.3:44589/ using document root <none> and password file <none>
I20260929 20:46:41.017814 1859 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:41.017902 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:41.018186 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:41.020876 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "07b852f019974bbf97d358e63323f43e"
format_stamp: "Formatted at 2026-09-29 20:46:41 on dist-test-slave-blv7"
server_key: "519a2d2fbdc03610e3526e0cc49d039d"
server_key_iv: "627b322ea4ba8f6df93c004511264346"
server_key_version: "encryptionkey@0"
I20260929 20:46:41.021597 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "07b852f019974bbf97d358e63323f43e"
format_stamp: "Formatted at 2026-09-29 20:46:41 on dist-test-slave-blv7"
server_key: "519a2d2fbdc03610e3526e0cc49d039d"
server_key_iv: "627b322ea4ba8f6df93c004511264346"
server_key_version: "encryptionkey@0"
I20260929 20:46:41.026221 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260929 20:46:41.029371 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:41.030598 1859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 20:46:41.030726 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "07b852f019974bbf97d358e63323f43e"
format_stamp: "Formatted at 2026-09-29 20:46:41 on dist-test-slave-blv7"
server_key: "519a2d2fbdc03610e3526e0cc49d039d"
server_key_iv: "627b322ea4ba8f6df93c004511264346"
server_key_version: "encryptionkey@0"
I20260929 20:46:41.030922 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:41.048576 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:41.052634 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:41.052861 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:41.055265 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:46:41.055357 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:41.055482 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:46:41.055549 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:46:41 dist-test-slave-blv7 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:41.077266 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.055981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40983 (local address 127.0.97.62:45269)
0929 20:46:41.056158 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:41.056163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:41.057536 (+ 1373us) server_negotiation.cc:407] Connection header received
0929 20:46:41.059583 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:41.059589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:41.059676 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:41.059787 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:41.062203 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.063405 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.065601 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.066150 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.066264 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.070652 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:41.070657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:41.070660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:41.070689 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:41.074161 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:41.074830 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:41.074835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:41.074839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:41.074990 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:41.075469 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:41.075475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:41.075478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:41.075792 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:41.075922 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:41.076867 (+ 945us) server_negotiation.cc:299] Negotiation successful
0929 20:46:41.077030 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:46:41.078778 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.056579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45269 (local address 127.0.97.3:40983)
0929 20:46:41.057381 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:41.057420 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:41.059265 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:41.060214 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:41.060230 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:41.061024 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:41.061965 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.061997 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.063582 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.063587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.065456 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.065474 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.066382 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.066390 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.066646 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.067400 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:41.067430 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:41.070438 (+ 3008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:41.074284 (+ 3846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:41.074309 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:41.074350 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:41.074731 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:41.075112 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:41.075117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:41.075120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:41.075370 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:41.076007 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:41.076018 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:41.076459 (+ 441us) client_negotiation.cc:769] Sending connection context
0929 20:46:41.076755 (+ 296us) client_negotiation.cc:240] Negotiation successful
0929 20:46:41.077279 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":334,"threads_started":1}
I20260929 20:46:41.097997 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.083980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54961 (local address 127.0.97.62:45269)
0929 20:46:41.084143 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:41.084148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:41.084751 (+ 603us) server_negotiation.cc:407] Connection header received
0929 20:46:41.084916 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:41.084921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:41.084992 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:41.085083 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:41.086402 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.087276 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.089589 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.090044 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.090136 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.092575 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:41.092581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:41.092586 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:41.092623 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:41.095492 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:41.096222 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:41.096228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:41.096232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:41.096378 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:41.096710 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:41.096715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:41.096719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:41.096951 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:41.097083 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:41.097451 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 20:46:41.097634 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:46:41.098069 1949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.084004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45269 (local address 127.0.97.3:54961)
0929 20:46:41.084655 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:41.084673 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:41.084805 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:41.085143 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:41.085149 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:41.085801 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:41.086256 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.086270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.087444 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.087450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.089441 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.089460 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.090224 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.090231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.090411 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.091159 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:41.091182 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:41.092409 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:41.095603 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:41.095609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:41.095616 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:41.096120 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:41.096478 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:41.096482 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:41.096486 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:41.096613 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:41.097057 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:41.097062 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:41.097222 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 20:46:41.097336 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 20:46:41.097636 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":377,"threads_started":1}
I20260929 20:46:41.100054 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42149
I20260929 20:46:41.100075 1993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42149 every 8 connection(s)
I20260929 20:46:41.101286 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:46:41.101732 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 1859
I20260929 20:46:41.101914 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:46:41.102487 388 external_mini_cluster.cc:1428] Setting key 7bb0070597ea1c3ac9784426eeb729b7
I20260929 20:46:41.103008 1994 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45269
I20260929 20:46:41.103219 1994 heartbeater.cc:464] Registering TS with master...
I20260929 20:46:41.104283 1994 heartbeater.cc:511] Master 127.0.97.62:45269 requested a full tablet report, sending...
I20260929 20:46:41.105537 1510 ts_manager.cc:194] Registered new tserver with Master: 07b852f019974bbf97d358e63323f43e (127.0.97.3:42149)
I20260929 20:46:41.106261 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40983
I20260929 20:46:41.117221 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:46:41 dist-test-slave-blv7 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714801, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:46:41 dist-test-slave-blv7 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714801, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:41.157248 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.144375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47510 (local address 127.0.97.62:45269)
0929 20:46:41.144590 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:41.144597 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:41.144730 (+ 133us) server_negotiation.cc:407] Connection header received
0929 20:46:41.144962 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:41.144967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:41.145030 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:41.145111 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:41.146530 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.147380 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.148578 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.149004 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.149109 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.152701 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:41.152705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:41.152708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:41.152742 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:41.154983 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:41.155541 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:41.155547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:41.155550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:41.155660 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:41.155996 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:41.156001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:41.156004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:41.156259 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:41.156480 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:41.156770 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 20:46:41.156967 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 20:46:41.161315 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:46:41.167891 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:46:41.188226 1716 heartbeater.cc:503] Master 127.0.97.62:45269 was elected leader, sending a full tablet report...
I20260929 20:46:41.198997 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.190433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44427 (local address 127.0.0.1:57822)
0929 20:46:41.191480 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:41.191500 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:41.191599 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:41.192577 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:41.192581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:41.192603 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:46:41.192982 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.192994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.194752 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.194757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.196957 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.196966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.198070 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.198076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.198243 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.198302 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 20:46:41.198429 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 20:46:41.198602 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"mutex_wait_us":49,"thread_start_us":792,"threads_started":1}
I20260929 20:46:41.199183 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.178940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42149 (local address 127.0.0.1:39452)
0929 20:46:41.187562 (+ 8622us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:41.187597 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:41.187768 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:41.192030 (+ 4262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:41.192037 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:41.192066 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:46:41.192486 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.192499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.194282 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.194287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.196684 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.196698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.198408 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.198414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.198573 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.198602 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 20:46:41.198666 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 20:46:41.198789 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8394,"mutex_wait_us":84,"thread_start_us":8256,"threads_started":1}
I20260929 20:46:41.199901 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.188217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57822 (local address 127.0.97.2:44427)
0929 20:46:41.192019 (+ 3802us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:41.192026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:41.192051 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:46:41.192139 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:41.192145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:41.192377 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:41.193152 (+ 775us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:46:41.193298 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.194617 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.197081 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.197902 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.198014 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.198201 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:41.199517 (+ 1316us) server_negotiation.cc:299] Negotiation successful
0929 20:46:41.199677 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3680,"thread_start_us":3563,"threads_started":1}
I20260929 20:46:41.199994 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.179381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39452 (local address 127.0.97.3:42149)
0929 20:46:41.191226 (+ 11845us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:41.191234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:41.191268 (+ 34us) server_negotiation.cc:407] Connection header received
0929 20:46:41.191377 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:41.191385 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:41.191627 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:41.192687 (+ 1060us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:46:41.192962 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.194156 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.196835 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.198212 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.198356 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.199472 (+ 1116us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:41.199545 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 20:46:41.199696 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11678,"thread_start_us":11509,"threads_started":1}
I20260929 20:46:41.200887 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.179586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51496 (local address 127.0.97.1:41333)
0929 20:46:41.191345 (+ 11759us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:41.191352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:41.191372 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:46:41.191447 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:41.191453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:41.191625 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:41.191804 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:46:41.193555 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.194887 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.197156 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.198470 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.198576 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.198744 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:41.199728 (+ 984us) server_negotiation.cc:299] Negotiation successful
0929 20:46:41.199880 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11634,"thread_start_us":12173,"threads_started":1}
I20260929 20:46:41.201393 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.179379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41333 (local address 127.0.0.1:51496)
0929 20:46:41.190829 (+ 11450us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:41.190848 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:41.190958 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:41.193021 (+ 2063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:41.193025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:41.193051 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:46:41.193379 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.193390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.195090 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.195095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.196981 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.196994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.198893 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.198900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.199120 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.199155 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 20:46:41.199236 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 20:46:41.199345 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11310,"mutex_wait_us":26,"thread_start_us":12575,"threads_started":1}
I20260929 20:46:41.203186 1785 tablet_service.cc:1467] Processing CreateTablet for tablet 762a3fc2f9874b36b0728167a14822ec (DEFAULT_TABLE table=test-table [id=4ca4eb52d2b24d858cfd601f00c7107b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:46:41.203197 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 762a3fc2f9874b36b0728167a14822ec (DEFAULT_TABLE table=test-table [id=4ca4eb52d2b24d858cfd601f00c7107b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:46:41.204424 1785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 762a3fc2f9874b36b0728167a14822ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:41.204519 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 762a3fc2f9874b36b0728167a14822ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:41.204435 1646 tablet_service.cc:1467] Processing CreateTablet for tablet 762a3fc2f9874b36b0728167a14822ec (DEFAULT_TABLE table=test-table [id=4ca4eb52d2b24d858cfd601f00c7107b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:46:41.205591 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 762a3fc2f9874b36b0728167a14822ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:41.214679 2013 tablet_bootstrap.cc:492] T 762a3fc2f9874b36b0728167a14822ec P 63969b25b0244237b7729b899aea297c: Bootstrap starting.
I20260929 20:46:41.218222 2013 tablet_bootstrap.cc:654] T 762a3fc2f9874b36b0728167a14822ec P 63969b25b0244237b7729b899aea297c: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:41.218210 2014 tablet_bootstrap.cc:492] T 762a3fc2f9874b36b0728167a14822ec P 07b852f019974bbf97d358e63323f43e: Bootstrap starting.
I20260929 20:46:41.219561 2013 log.cc:824] T 762a3fc2f9874b36b0728167a14822ec P 63969b25b0244237b7729b899aea297c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:46:41.219612 2015 tablet_bootstrap.cc:492] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a: Bootstrap starting.
I20260929 20:46:41.221361 2014 tablet_bootstrap.cc:654] T 762a3fc2f9874b36b0728167a14822ec P 07b852f019974bbf97d358e63323f43e: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:41.222530 2014 log.cc:824] T 762a3fc2f9874b36b0728167a14822ec P 07b852f019974bbf97d358e63323f43e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:46:41.222874 2013 tablet_bootstrap.cc:492] T 762a3fc2f9874b36b0728167a14822ec P 63969b25b0244237b7729b899aea297c: No bootstrap required, opened a new log
I20260929 20:46:41.223171 2015 tablet_bootstrap.cc:654] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:41.223167 2013 ts_tablet_manager.cc:1397] T 762a3fc2f9874b36b0728167a14822ec P 63969b25b0244237b7729b899aea297c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 20:46:41.224169 2015 log.cc:824] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:46:41.225323 2014 tablet_bootstrap.cc:492] T 762a3fc2f9874b36b0728167a14822ec P 07b852f019974bbf97d358e63323f43e: No bootstrap required, opened a new log
I20260929 20:46:41.225553 2014 ts_tablet_manager.cc:1397] T 762a3fc2f9874b36b0728167a14822ec P 07b852f019974bbf97d358e63323f43e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260929 20:46:41.226298 2015 tablet_bootstrap.cc:492] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a: No bootstrap required, opened a new log
I20260929 20:46:41.226505 2015 ts_tablet_manager.cc:1397] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260929 20:46:41.232940 2013 raft_consensus.cc:348] T 762a3fc2f9874b36b0728167a14822ec P 63969b25b0244237b7729b899aea297c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } }
I20260929 20:46:41.233397 2013 raft_consensus.cc:374] T 762a3fc2f9874b36b0728167a14822ec P 63969b25b0244237b7729b899aea297c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:41.233541 2013 raft_consensus.cc:729] T 762a3fc2f9874b36b0728167a14822ec P 63969b25b0244237b7729b899aea297c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63969b25b0244237b7729b899aea297c, State: Initialized, Role: FOLLOWER
I20260929 20:46:41.233872 2015 raft_consensus.cc:348] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } }
I20260929 20:46:41.234218 2015 raft_consensus.cc:374] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:41.234311 2015 raft_consensus.cc:729] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2c783b8c4c401ba731af02a500a22a, State: Initialized, Role: FOLLOWER
I20260929 20:46:41.234684 2013 consensus_queue.cc:260] T 762a3fc2f9874b36b0728167a14822ec P 63969b25b0244237b7729b899aea297c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } }
I20260929 20:46:41.234823 2015 consensus_queue.cc:260] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } }
I20260929 20:46:41.235606 2014 raft_consensus.cc:348] T 762a3fc2f9874b36b0728167a14822ec P 07b852f019974bbf97d358e63323f43e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } }
I20260929 20:46:41.235988 2014 raft_consensus.cc:374] T 762a3fc2f9874b36b0728167a14822ec P 07b852f019974bbf97d358e63323f43e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:41.236086 2014 raft_consensus.cc:729] T 762a3fc2f9874b36b0728167a14822ec P 07b852f019974bbf97d358e63323f43e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b852f019974bbf97d358e63323f43e, State: Initialized, Role: FOLLOWER
I20260929 20:46:41.236515 2015 ts_tablet_manager.cc:1428] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260929 20:46:41.236629 2014 consensus_queue.cc:260] T 762a3fc2f9874b36b0728167a14822ec P 07b852f019974bbf97d358e63323f43e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } }
I20260929 20:46:41.237599 1994 heartbeater.cc:503] Master 127.0.97.62:45269 was elected leader, sending a full tablet report...
I20260929 20:46:41.238252 2014 ts_tablet_manager.cc:1428] T 762a3fc2f9874b36b0728167a14822ec P 07b852f019974bbf97d358e63323f43e: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
I20260929 20:46:41.239120 1855 heartbeater.cc:503] Master 127.0.97.62:45269 was elected leader, sending a full tablet report...
I20260929 20:46:41.240543 2013 ts_tablet_manager.cc:1428] T 762a3fc2f9874b36b0728167a14822ec P 63969b25b0244237b7729b899aea297c: Time spent starting tablet: real 0.017s user 0.010s sys 0.004s
I20260929 20:46:41.290668 2019 raft_consensus.cc:482] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:46:41.290933 2019 raft_consensus.cc:504] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } }
I20260929 20:46:41.292541 2019 leader_election.cc:288] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07b852f019974bbf97d358e63323f43e (127.0.97.3:42149), 63969b25b0244237b7729b899aea297c (127.0.97.2:44427)
I20260929 20:46:41.299063 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.292894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42149 (local address 127.0.97.1:35065)
0929 20:46:41.293504 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:41.293521 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:41.293651 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:41.294148 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:41.294153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:41.294180 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:46:41.294906 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.294919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.295868 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.295874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.297292 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.297306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.298386 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.298392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.298507 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.298530 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:46:41.298617 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 20:46:41.298733 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":333,"threads_started":1}
I20260929 20:46:41.299533 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.292897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35065 (local address 127.0.97.3:42149)
0929 20:46:41.293098 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:41.293102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:41.293629 (+ 527us) server_negotiation.cc:407] Connection header received
0929 20:46:41.293879 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:41.293885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:41.293964 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:41.294057 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:46:41.295070 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.295744 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.297432 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.298178 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.298285 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.298384 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:41.298652 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 20:46:41.298786 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:46:41.299880 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.293880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46309 (local address 127.0.97.2:44427)
0929 20:46:41.294063 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:41.294068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:41.295048 (+ 980us) server_negotiation.cc:407] Connection header received
0929 20:46:41.295193 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:41.295199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:41.295260 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:41.295338 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:46:41.295794 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.296505 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.298038 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.298959 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.299113 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.299202 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:41.299532 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 20:46:41.299648 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:46:41.299911 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.293648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44427 (local address 127.0.97.1:46309)
0929 20:46:41.294927 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:41.294940 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:41.295074 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:41.295358 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:41.295362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:41.295384 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:46:41.295658 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.295666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.296636 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.296639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.297892 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.297903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.299180 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.299187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.299344 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.299374 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:46:41.299493 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 20:46:41.299602 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"thread_start_us":966,"threads_started":1}
I20260929 20:46:41.300383 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "762a3fc2f9874b36b0728167a14822ec" candidate_uuid: "de2c783b8c4c401ba731af02a500a22a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b852f019974bbf97d358e63323f43e" is_pre_election: true
I20260929 20:46:41.300853 1948 raft_consensus.cc:2454] T 762a3fc2f9874b36b0728167a14822ec P 07b852f019974bbf97d358e63323f43e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de2c783b8c4c401ba731af02a500a22a in term 0.
I20260929 20:46:41.300827 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "762a3fc2f9874b36b0728167a14822ec" candidate_uuid: "de2c783b8c4c401ba731af02a500a22a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63969b25b0244237b7729b899aea297c" is_pre_election: true
I20260929 20:46:41.301262 1806 raft_consensus.cc:2454] T 762a3fc2f9874b36b0728167a14822ec P 63969b25b0244237b7729b899aea297c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de2c783b8c4c401ba731af02a500a22a in term 0.
I20260929 20:46:41.301424 1598 leader_election.cc:302] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b852f019974bbf97d358e63323f43e, de2c783b8c4c401ba731af02a500a22a; no voters:
I20260929 20:46:41.301787 2019 raft_consensus.cc:2781] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:46:41.301925 2019 raft_consensus.cc:482] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:46:41.302029 2019 raft_consensus.cc:3037] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:41.304124 2019 raft_consensus.cc:504] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } }
I20260929 20:46:41.304673 2019 leader_election.cc:288] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a [CANDIDATE]: Term 1 election: Requested vote from peers 07b852f019974bbf97d358e63323f43e (127.0.97.3:42149), 63969b25b0244237b7729b899aea297c (127.0.97.2:44427)
I20260929 20:46:41.305071 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "762a3fc2f9874b36b0728167a14822ec" candidate_uuid: "de2c783b8c4c401ba731af02a500a22a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63969b25b0244237b7729b899aea297c"
I20260929 20:46:41.305064 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "762a3fc2f9874b36b0728167a14822ec" candidate_uuid: "de2c783b8c4c401ba731af02a500a22a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b852f019974bbf97d358e63323f43e"
I20260929 20:46:41.305229 1806 raft_consensus.cc:3037] T 762a3fc2f9874b36b0728167a14822ec P 63969b25b0244237b7729b899aea297c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:41.305227 1948 raft_consensus.cc:3037] T 762a3fc2f9874b36b0728167a14822ec P 07b852f019974bbf97d358e63323f43e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:41.307696 1806 raft_consensus.cc:2454] T 762a3fc2f9874b36b0728167a14822ec P 63969b25b0244237b7729b899aea297c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de2c783b8c4c401ba731af02a500a22a in term 1.
I20260929 20:46:41.307797 1948 raft_consensus.cc:2454] T 762a3fc2f9874b36b0728167a14822ec P 07b852f019974bbf97d358e63323f43e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de2c783b8c4c401ba731af02a500a22a in term 1.
I20260929 20:46:41.308019 1598 leader_election.cc:302] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63969b25b0244237b7729b899aea297c, de2c783b8c4c401ba731af02a500a22a; no voters:
I20260929 20:46:41.308298 2019 raft_consensus.cc:2781] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:46:41.308667 2019 raft_consensus.cc:686] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a [term 1 LEADER]: Becoming Leader. State: Replica: de2c783b8c4c401ba731af02a500a22a, State: Running, Role: LEADER
I20260929 20:46:41.309098 2019 consensus_queue.cc:237] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } }
I20260929 20:46:41.313436 1509 catalog_manager.cc:5495] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a reported cstate change: term changed from 0 to 1, leader changed from <none> to de2c783b8c4c401ba731af02a500a22a (127.0.97.1). New cstate: current_term: 1 leader_uuid: "de2c783b8c4c401ba731af02a500a22a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } health_report { overall_health: HEALTHY } } }
I20260929 20:46:41.339562 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.334574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51506 (local address 127.0.97.1:41333)
0929 20:46:41.334784 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:41.334791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:41.334911 (+ 120us) server_negotiation.cc:407] Connection header received
0929 20:46:41.335242 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:41.335246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:41.335341 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:41.335463 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:46:41.335766 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.336441 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.337664 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.338053 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.338141 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.338586 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:46:41.339073 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:46:41.339177 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:41.339312 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 20:46:41.339371 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:46:41.347959 1948 raft_consensus.cc:1261] T 762a3fc2f9874b36b0728167a14822ec P 07b852f019974bbf97d358e63323f43e [term 1 FOLLOWER]: Refusing update from remote peer de2c783b8c4c401ba731af02a500a22a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:46:41.347976 1806 raft_consensus.cc:1261] T 762a3fc2f9874b36b0728167a14822ec P 63969b25b0244237b7729b899aea297c [term 1 FOLLOWER]: Refusing update from remote peer de2c783b8c4c401ba731af02a500a22a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:46:41.348937 2024 consensus_queue.cc:1035] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:46:41.349256 2019 consensus_queue.cc:1035] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 20:46:41.354523 1995 tablet.cc:2369] T 762a3fc2f9874b36b0728167a14822ec P 07b852f019974bbf97d358e63323f43e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:46:41.362653 2029 mvcc.cc:205] Tried to move back new op lower bound from 7334767826311172096 to 7334767826165481472. Current Snapshot: MvccSnapshot[applied={T|T < 7334767826311172096}]
I20260929 20:46:41.377557 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:46:41.377790 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:46:41.380860 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476: Sending DeleteTablet for 3 replicas of tablet 762a3fc2f9874b36b0728167a14822ec
I20260929 20:46:41.381825 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 762a3fc2f9874b36b0728167a14822ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:46:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39452
I20260929 20:46:41.381917 1785 tablet_service.cc:1514] Processing DeleteTablet for tablet 762a3fc2f9874b36b0728167a14822ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:46:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57822
I20260929 20:46:41.382483 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet 762a3fc2f9874b36b0728167a14822ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:46:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51496
I20260929 20:46:41.383567 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:46:41.384186 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:46:41.392657 1645 tablet_service.cc:1467] Processing CreateTablet for tablet c39795da2f784bf592b69cd890fa0301 (DEFAULT_TABLE table=test-table [id=91f5599254d344608509decb3e87392e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:46:41.393203 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c39795da2f784bf592b69cd890fa0301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:41.393118 1784 tablet_service.cc:1467] Processing CreateTablet for tablet c39795da2f784bf592b69cd890fa0301 (DEFAULT_TABLE table=test-table [id=91f5599254d344608509decb3e87392e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:46:41.393683 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c39795da2f784bf592b69cd890fa0301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:41.393715 2038 tablet_replica.cc:330] T 762a3fc2f9874b36b0728167a14822ec P 63969b25b0244237b7729b899aea297c: stopping tablet replica
I20260929 20:46:41.394132 2038 raft_consensus.cc:2229] T 762a3fc2f9874b36b0728167a14822ec P 63969b25b0244237b7729b899aea297c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:46:41.394662 2038 raft_consensus.cc:2258] T 762a3fc2f9874b36b0728167a14822ec P 63969b25b0244237b7729b899aea297c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:46:41.395346 2037 tablet_replica.cc:330] T 762a3fc2f9874b36b0728167a14822ec P 07b852f019974bbf97d358e63323f43e: stopping tablet replica
I20260929 20:46:41.395670 2037 raft_consensus.cc:2229] T 762a3fc2f9874b36b0728167a14822ec P 07b852f019974bbf97d358e63323f43e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:46:41.396593 2037 raft_consensus.cc:2258] T 762a3fc2f9874b36b0728167a14822ec P 07b852f019974bbf97d358e63323f43e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:46:41.399422 1923 tablet_service.cc:1467] Processing CreateTablet for tablet c39795da2f784bf592b69cd890fa0301 (DEFAULT_TABLE table=test-table [id=91f5599254d344608509decb3e87392e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:46:41.399775 2013 tablet_bootstrap.cc:492] T c39795da2f784bf592b69cd890fa0301 P 63969b25b0244237b7729b899aea297c: Bootstrap starting.
I20260929 20:46:41.399832 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c39795da2f784bf592b69cd890fa0301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:41.400985 2013 tablet_bootstrap.cc:654] T c39795da2f784bf592b69cd890fa0301 P 63969b25b0244237b7729b899aea297c: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:41.402619 2013 tablet_bootstrap.cc:492] T c39795da2f784bf592b69cd890fa0301 P 63969b25b0244237b7729b899aea297c: No bootstrap required, opened a new log
I20260929 20:46:41.402740 2013 ts_tablet_manager.cc:1397] T c39795da2f784bf592b69cd890fa0301 P 63969b25b0244237b7729b899aea297c: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260929 20:46:41.403258 2039 tablet_replica.cc:330] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a: stopping tablet replica
I20260929 20:46:41.403476 2013 raft_consensus.cc:348] T c39795da2f784bf592b69cd890fa0301 P 63969b25b0244237b7729b899aea297c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } }
I20260929 20:46:41.403671 2013 raft_consensus.cc:374] T c39795da2f784bf592b69cd890fa0301 P 63969b25b0244237b7729b899aea297c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:41.403638 2039 raft_consensus.cc:2229] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:46:41.403745 2013 raft_consensus.cc:729] T c39795da2f784bf592b69cd890fa0301 P 63969b25b0244237b7729b899aea297c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63969b25b0244237b7729b899aea297c, State: Initialized, Role: FOLLOWER
I20260929 20:46:41.404052 2013 consensus_queue.cc:260] T c39795da2f784bf592b69cd890fa0301 P 63969b25b0244237b7729b899aea297c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } }
I20260929 20:46:41.404214 2039 raft_consensus.cc:2258] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:46:41.404657 2013 ts_tablet_manager.cc:1428] T c39795da2f784bf592b69cd890fa0301 P 63969b25b0244237b7729b899aea297c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:46:41.405455 2014 tablet_bootstrap.cc:492] T c39795da2f784bf592b69cd890fa0301 P 07b852f019974bbf97d358e63323f43e: Bootstrap starting.
I20260929 20:46:41.406804 2037 ts_tablet_manager.cc:1905] T 762a3fc2f9874b36b0728167a14822ec P 07b852f019974bbf97d358e63323f43e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:46:41.407480 2014 tablet_bootstrap.cc:654] T c39795da2f784bf592b69cd890fa0301 P 07b852f019974bbf97d358e63323f43e: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:41.407661 2038 ts_tablet_manager.cc:1905] T 762a3fc2f9874b36b0728167a14822ec P 63969b25b0244237b7729b899aea297c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:46:41.408960 2015 tablet_bootstrap.cc:492] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a: Bootstrap starting.
I20260929 20:46:41.410274 2015 tablet_bootstrap.cc:654] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:41.411567 2014 tablet_bootstrap.cc:492] T c39795da2f784bf592b69cd890fa0301 P 07b852f019974bbf97d358e63323f43e: No bootstrap required, opened a new log
I20260929 20:46:41.411695 2014 ts_tablet_manager.cc:1397] T c39795da2f784bf592b69cd890fa0301 P 07b852f019974bbf97d358e63323f43e: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 20:46:41.412321 2014 raft_consensus.cc:348] T c39795da2f784bf592b69cd890fa0301 P 07b852f019974bbf97d358e63323f43e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } }
I20260929 20:46:41.412479 2014 raft_consensus.cc:374] T c39795da2f784bf592b69cd890fa0301 P 07b852f019974bbf97d358e63323f43e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:41.412569 2014 raft_consensus.cc:729] T c39795da2f784bf592b69cd890fa0301 P 07b852f019974bbf97d358e63323f43e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b852f019974bbf97d358e63323f43e, State: Initialized, Role: FOLLOWER
I20260929 20:46:41.412820 2014 consensus_queue.cc:260] T c39795da2f784bf592b69cd890fa0301 P 07b852f019974bbf97d358e63323f43e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } }
I20260929 20:46:41.414690 2038 ts_tablet_manager.cc:1918] T 762a3fc2f9874b36b0728167a14822ec P 63969b25b0244237b7729b899aea297c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:46:41.414830 2038 log.cc:1196] T 762a3fc2f9874b36b0728167a14822ec P 63969b25b0244237b7729b899aea297c: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/762a3fc2f9874b36b0728167a14822ec
I20260929 20:46:41.415447 2038 ts_tablet_manager.cc:1939] T 762a3fc2f9874b36b0728167a14822ec P 63969b25b0244237b7729b899aea297c: Deleting consensus metadata
I20260929 20:46:41.415432 2014 ts_tablet_manager.cc:1428] T c39795da2f784bf592b69cd890fa0301 P 07b852f019974bbf97d358e63323f43e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 20:46:41.415642 2037 ts_tablet_manager.cc:1918] T 762a3fc2f9874b36b0728167a14822ec P 07b852f019974bbf97d358e63323f43e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:46:41.415802 2037 log.cc:1196] T 762a3fc2f9874b36b0728167a14822ec P 07b852f019974bbf97d358e63323f43e: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/762a3fc2f9874b36b0728167a14822ec
I20260929 20:46:41.416327 2037 ts_tablet_manager.cc:1939] T 762a3fc2f9874b36b0728167a14822ec P 07b852f019974bbf97d358e63323f43e: Deleting consensus metadata
I20260929 20:46:41.416558 2039 ts_tablet_manager.cc:1905] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:46:41.417152 1495 catalog_manager.cc:4841] TS 63969b25b0244237b7729b899aea297c (127.0.97.2:44427): tablet 762a3fc2f9874b36b0728167a14822ec (table test-table [id=4ca4eb52d2b24d858cfd601f00c7107b]) successfully deleted
I20260929 20:46:41.417972 1495 catalog_manager.cc:4841] TS 07b852f019974bbf97d358e63323f43e (127.0.97.3:42149): tablet 762a3fc2f9874b36b0728167a14822ec (table test-table [id=4ca4eb52d2b24d858cfd601f00c7107b]) successfully deleted
I20260929 20:46:41.419818 2015 tablet_bootstrap.cc:492] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a: No bootstrap required, opened a new log
I20260929 20:46:41.419991 2015 ts_tablet_manager.cc:1397] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260929 20:46:41.420994 2015 raft_consensus.cc:348] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } }
I20260929 20:46:41.421238 2015 raft_consensus.cc:374] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:41.421386 2015 raft_consensus.cc:729] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2c783b8c4c401ba731af02a500a22a, State: Initialized, Role: FOLLOWER
I20260929 20:46:41.421662 2015 consensus_queue.cc:260] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } }
I20260929 20:46:41.422008 2039 ts_tablet_manager.cc:1918] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:46:41.422168 2039 log.cc:1196] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/762a3fc2f9874b36b0728167a14822ec
I20260929 20:46:41.422299 2015 ts_tablet_manager.cc:1428] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:46:41.422685 2039 ts_tablet_manager.cc:1939] T 762a3fc2f9874b36b0728167a14822ec P de2c783b8c4c401ba731af02a500a22a: Deleting consensus metadata
I20260929 20:46:41.424260 1496 catalog_manager.cc:4841] TS de2c783b8c4c401ba731af02a500a22a (127.0.97.1:41333): tablet 762a3fc2f9874b36b0728167a14822ec (table test-table [id=4ca4eb52d2b24d858cfd601f00c7107b]) successfully deleted
W20260929 20:46:41.431746 1717 tablet.cc:2369] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:46:41.639676 1856 tablet.cc:2369] T c39795da2f784bf592b69cd890fa0301 P 63969b25b0244237b7729b899aea297c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:46:41.661230 2033 raft_consensus.cc:482] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:46:41.661566 2033 raft_consensus.cc:504] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } }
I20260929 20:46:41.662576 2033 leader_election.cc:288] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63969b25b0244237b7729b899aea297c (127.0.97.2:44427), 07b852f019974bbf97d358e63323f43e (127.0.97.3:42149)
I20260929 20:46:41.663067 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39795da2f784bf592b69cd890fa0301" candidate_uuid: "de2c783b8c4c401ba731af02a500a22a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63969b25b0244237b7729b899aea297c" is_pre_election: true
I20260929 20:46:41.663197 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39795da2f784bf592b69cd890fa0301" candidate_uuid: "de2c783b8c4c401ba731af02a500a22a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b852f019974bbf97d358e63323f43e" is_pre_election: true
I20260929 20:46:41.663316 1806 raft_consensus.cc:2454] T c39795da2f784bf592b69cd890fa0301 P 63969b25b0244237b7729b899aea297c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de2c783b8c4c401ba731af02a500a22a in term 0.
I20260929 20:46:41.663440 1948 raft_consensus.cc:2454] T c39795da2f784bf592b69cd890fa0301 P 07b852f019974bbf97d358e63323f43e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de2c783b8c4c401ba731af02a500a22a in term 0.
I20260929 20:46:41.663763 1598 leader_election.cc:302] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63969b25b0244237b7729b899aea297c, de2c783b8c4c401ba731af02a500a22a; no voters:
I20260929 20:46:41.664021 2033 raft_consensus.cc:2781] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:46:41.664181 2033 raft_consensus.cc:482] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:46:41.664304 2033 raft_consensus.cc:3037] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:41.667575 2033 raft_consensus.cc:504] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } }
I20260929 20:46:41.668319 2033 leader_election.cc:288] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a [CANDIDATE]: Term 1 election: Requested vote from peers 63969b25b0244237b7729b899aea297c (127.0.97.2:44427), 07b852f019974bbf97d358e63323f43e (127.0.97.3:42149)
I20260929 20:46:41.668773 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39795da2f784bf592b69cd890fa0301" candidate_uuid: "de2c783b8c4c401ba731af02a500a22a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63969b25b0244237b7729b899aea297c"
I20260929 20:46:41.668838 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c39795da2f784bf592b69cd890fa0301" candidate_uuid: "de2c783b8c4c401ba731af02a500a22a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b852f019974bbf97d358e63323f43e"
I20260929 20:46:41.669009 1948 raft_consensus.cc:3037] T c39795da2f784bf592b69cd890fa0301 P 07b852f019974bbf97d358e63323f43e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:41.669003 1806 raft_consensus.cc:3037] T c39795da2f784bf592b69cd890fa0301 P 63969b25b0244237b7729b899aea297c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:41.670864 1948 raft_consensus.cc:2454] T c39795da2f784bf592b69cd890fa0301 P 07b852f019974bbf97d358e63323f43e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de2c783b8c4c401ba731af02a500a22a in term 1.
I20260929 20:46:41.671253 1598 leader_election.cc:302] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b852f019974bbf97d358e63323f43e, de2c783b8c4c401ba731af02a500a22a; no voters:
I20260929 20:46:41.671509 2033 raft_consensus.cc:2781] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:46:41.671655 2033 raft_consensus.cc:686] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a [term 1 LEADER]: Becoming Leader. State: Replica: de2c783b8c4c401ba731af02a500a22a, State: Running, Role: LEADER
I20260929 20:46:41.671973 1806 raft_consensus.cc:2454] T c39795da2f784bf592b69cd890fa0301 P 63969b25b0244237b7729b899aea297c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de2c783b8c4c401ba731af02a500a22a in term 1.
I20260929 20:46:41.671926 2033 consensus_queue.cc:237] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } }
I20260929 20:46:41.674561 1508 catalog_manager.cc:5495] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a reported cstate change: term changed from 0 to 1, leader changed from <none> to de2c783b8c4c401ba731af02a500a22a (127.0.97.1). New cstate: current_term: 1 leader_uuid: "de2c783b8c4c401ba731af02a500a22a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } health_report { overall_health: UNKNOWN } } }
Sep 29 20:46:41 dist-test-slave-blv7 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:41.751754 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.737128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45269 (local address 127.0.0.1:47524)
0929 20:46:41.739617 (+ 2489us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:41.739636 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:41.739752 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:41.740094 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:41.740097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:41.740730 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:41.740950 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.740959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.742021 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.742026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.742942 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.742953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.743606 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.743611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.743763 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.744343 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:41.744360 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:41.746631 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:41.749453 (+ 2822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:41.749465 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:41.749473 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:41.749853 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:41.750156 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:41.750161 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:41.750165 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:41.750302 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:41.750765 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:41.750773 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:41.750919 (+ 146us) client_negotiation.cc:769] Sending connection context
0929 20:46:41.751066 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 20:46:41.751317 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2366,"thread_start_us":2241,"threads_started":1}
I20260929 20:46:41.751808 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.737230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47524 (local address 127.0.97.62:45269)
0929 20:46:41.739457 (+ 2227us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:41.739463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:41.739719 (+ 256us) server_negotiation.cc:407] Connection header received
0929 20:46:41.739864 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:41.739870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:41.739948 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:41.740076 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:41.741135 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.741902 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.743076 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.743466 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.743565 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.746824 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:41.746828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:41.746832 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:41.746858 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:41.749342 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:41.749950 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:41.749955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:41.749958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:41.750065 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:41.750394 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:41.750399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:41.750402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:41.750634 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:41.750755 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:41.751159 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 20:46:41.751438 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":2063,"thread_start_us":1949,"threads_started":1}
I20260929 20:46:41.756369 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47524:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 20:46:41.757339 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:46:41.764164 1645 tablet_service.cc:1467] Processing CreateTablet for tablet c674d49e75f24bd7b88c9808ddb764fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e2e91d5b25114b1e8eeaa608bf8c377d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:46:41.764268 1784 tablet_service.cc:1467] Processing CreateTablet for tablet c674d49e75f24bd7b88c9808ddb764fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e2e91d5b25114b1e8eeaa608bf8c377d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:46:41.764706 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c674d49e75f24bd7b88c9808ddb764fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:41.764547 1923 tablet_service.cc:1467] Processing CreateTablet for tablet c674d49e75f24bd7b88c9808ddb764fb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e2e91d5b25114b1e8eeaa608bf8c377d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:46:41.764793 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c674d49e75f24bd7b88c9808ddb764fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:41.765045 1923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c674d49e75f24bd7b88c9808ddb764fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:41.772149 2014 tablet_bootstrap.cc:492] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e: Bootstrap starting.
I20260929 20:46:41.772681 2013 tablet_bootstrap.cc:492] T c674d49e75f24bd7b88c9808ddb764fb P 63969b25b0244237b7729b899aea297c: Bootstrap starting.
I20260929 20:46:41.774070 2014 tablet_bootstrap.cc:654] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:41.774217 2013 tablet_bootstrap.cc:654] T c674d49e75f24bd7b88c9808ddb764fb P 63969b25b0244237b7729b899aea297c: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:41.775166 2015 tablet_bootstrap.cc:492] T c674d49e75f24bd7b88c9808ddb764fb P de2c783b8c4c401ba731af02a500a22a: Bootstrap starting.
I20260929 20:46:41.776784 2015 tablet_bootstrap.cc:654] T c674d49e75f24bd7b88c9808ddb764fb P de2c783b8c4c401ba731af02a500a22a: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:41.784155 2014 tablet_bootstrap.cc:492] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e: No bootstrap required, opened a new log
I20260929 20:46:41.784313 2014 ts_tablet_manager.cc:1397] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260929 20:46:41.784412 2013 tablet_bootstrap.cc:492] T c674d49e75f24bd7b88c9808ddb764fb P 63969b25b0244237b7729b899aea297c: No bootstrap required, opened a new log
I20260929 20:46:41.784572 2013 ts_tablet_manager.cc:1397] T c674d49e75f24bd7b88c9808ddb764fb P 63969b25b0244237b7729b899aea297c: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.004s
I20260929 20:46:41.785213 2013 raft_consensus.cc:348] T c674d49e75f24bd7b88c9808ddb764fb P 63969b25b0244237b7729b899aea297c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } }
I20260929 20:46:41.785282 2014 raft_consensus.cc:348] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } }
I20260929 20:46:41.785454 2013 raft_consensus.cc:374] T c674d49e75f24bd7b88c9808ddb764fb P 63969b25b0244237b7729b899aea297c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:41.785506 2014 raft_consensus.cc:374] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:41.785526 2013 raft_consensus.cc:729] T c674d49e75f24bd7b88c9808ddb764fb P 63969b25b0244237b7729b899aea297c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63969b25b0244237b7729b899aea297c, State: Initialized, Role: FOLLOWER
I20260929 20:46:41.785585 2014 raft_consensus.cc:729] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b852f019974bbf97d358e63323f43e, State: Initialized, Role: FOLLOWER
I20260929 20:46:41.785811 2014 consensus_queue.cc:260] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } }
I20260929 20:46:41.785816 2013 consensus_queue.cc:260] T c674d49e75f24bd7b88c9808ddb764fb P 63969b25b0244237b7729b899aea297c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } }
I20260929 20:46:41.786190 2021 tablet_replica.cc:439] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } } }
I20260929 20:46:41.786231 2020 tablet_replica.cc:439] T c674d49e75f24bd7b88c9808ddb764fb P 63969b25b0244237b7729b899aea297c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } } }
I20260929 20:46:41.786480 2021 tablet_replica.cc:442] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:46:41.786542 2020 tablet_replica.cc:442] T c674d49e75f24bd7b88c9808ddb764fb P 63969b25b0244237b7729b899aea297c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:46:41.786857 2014 ts_tablet_manager.cc:1428] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 20:46:41.787401 2013 ts_tablet_manager.cc:1428] T c674d49e75f24bd7b88c9808ddb764fb P 63969b25b0244237b7729b899aea297c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 20:46:41.788480 2015 tablet_bootstrap.cc:492] T c674d49e75f24bd7b88c9808ddb764fb P de2c783b8c4c401ba731af02a500a22a: No bootstrap required, opened a new log
I20260929 20:46:41.788715 2015 ts_tablet_manager.cc:1397] T c674d49e75f24bd7b88c9808ddb764fb P de2c783b8c4c401ba731af02a500a22a: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20260929 20:46:41.789812 2015 raft_consensus.cc:348] T c674d49e75f24bd7b88c9808ddb764fb P de2c783b8c4c401ba731af02a500a22a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } }
I20260929 20:46:41.790122 2015 raft_consensus.cc:374] T c674d49e75f24bd7b88c9808ddb764fb P de2c783b8c4c401ba731af02a500a22a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:41.790256 2015 raft_consensus.cc:729] T c674d49e75f24bd7b88c9808ddb764fb P de2c783b8c4c401ba731af02a500a22a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2c783b8c4c401ba731af02a500a22a, State: Initialized, Role: FOLLOWER
I20260929 20:46:41.791071 2015 consensus_queue.cc:260] T c674d49e75f24bd7b88c9808ddb764fb P de2c783b8c4c401ba731af02a500a22a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } }
I20260929 20:46:41.791460 2033 tablet_replica.cc:439] T c674d49e75f24bd7b88c9808ddb764fb P de2c783b8c4c401ba731af02a500a22a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } } }
I20260929 20:46:41.791620 2015 ts_tablet_manager.cc:1428] T c674d49e75f24bd7b88c9808ddb764fb P de2c783b8c4c401ba731af02a500a22a: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260929 20:46:41.791760 2033 tablet_replica.cc:442] T c674d49e75f24bd7b88c9808ddb764fb P de2c783b8c4c401ba731af02a500a22a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:46:41.802145 2021 raft_consensus.cc:482] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:46:41.802341 2021 raft_consensus.cc:504] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } }
I20260929 20:46:41.803982 2021 leader_election.cc:288] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de2c783b8c4c401ba731af02a500a22a (127.0.97.1:41333), 63969b25b0244237b7729b899aea297c (127.0.97.2:44427)
I20260929 20:46:41.821782 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.804442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38249 (local address 127.0.97.2:44427)
0929 20:46:41.815168 (+ 10726us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:41.815174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:41.815519 (+ 345us) server_negotiation.cc:407] Connection header received
0929 20:46:41.815646 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:41.815652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:41.815724 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:41.815811 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:46:41.816336 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.817484 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.819408 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.820818 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.820942 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.821039 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:41.821317 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 20:46:41.821447 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10273,"thread_start_us":10165,"threads_started":1}
I20260929 20:46:41.822279 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.804718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43665 (local address 127.0.97.1:41333)
0929 20:46:41.804901 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:41.804907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:41.815210 (+ 10303us) server_negotiation.cc:407] Connection header received
0929 20:46:41.815455 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:41.815460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:41.815539 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:41.815636 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:46:41.816414 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.817465 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.820067 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.821203 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.821323 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.821412 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:41.821879 (+ 467us) server_negotiation.cc:299] Negotiation successful
0929 20:46:41.822009 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:46:41.822573 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.804267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44427 (local address 127.0.97.3:38249)
0929 20:46:41.815432 (+ 11165us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:41.815447 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:41.815537 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:41.815872 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:41.815876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:41.815902 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:46:41.816184 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.816195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.817626 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.817631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.819257 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.819269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.820960 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.820967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.821149 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.821184 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 20:46:41.821271 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 20:46:41.822131 (+ 860us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11060,"mutex_wait_us":71,"thread_start_us":10587,"threads_started":1}
I20260929 20:46:41.823206 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c674d49e75f24bd7b88c9808ddb764fb" candidate_uuid: "07b852f019974bbf97d358e63323f43e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63969b25b0244237b7729b899aea297c" is_pre_election: true
I20260929 20:46:41.823462 1806 raft_consensus.cc:2454] T c674d49e75f24bd7b88c9808ddb764fb P 63969b25b0244237b7729b899aea297c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07b852f019974bbf97d358e63323f43e in term 0.
I20260929 20:46:41.823563 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.804181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41333 (local address 127.0.97.3:43665)
0929 20:46:41.815110 (+ 10929us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:41.815133 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:41.815319 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:41.815815 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:41.815820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:41.815846 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:46:41.816263 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.816275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.817578 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.817583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.819872 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.819886 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.821514 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.821520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.821687 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.821718 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:46:41.823191 (+ 1473us) client_negotiation.cc:240] Negotiation successful
0929 20:46:41.823287 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10783,"mutex_wait_us":36,"thread_start_us":10992,"threads_started":1}
I20260929 20:46:41.823951 1876 leader_election.cc:302] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b852f019974bbf97d358e63323f43e, 63969b25b0244237b7729b899aea297c; no voters:
I20260929 20:46:41.824335 2021 raft_consensus.cc:2781] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:46:41.824324 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c674d49e75f24bd7b88c9808ddb764fb" candidate_uuid: "07b852f019974bbf97d358e63323f43e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2c783b8c4c401ba731af02a500a22a" is_pre_election: true
I20260929 20:46:41.824445 2021 raft_consensus.cc:482] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:46:41.824528 2021 raft_consensus.cc:3037] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:41.824658 1671 raft_consensus.cc:2454] T c674d49e75f24bd7b88c9808ddb764fb P de2c783b8c4c401ba731af02a500a22a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07b852f019974bbf97d358e63323f43e in term 0.
I20260929 20:46:41.826716 2021 raft_consensus.cc:504] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } }
I20260929 20:46:41.827319 2021 leader_election.cc:288] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e [CANDIDATE]: Term 1 election: Requested vote from peers de2c783b8c4c401ba731af02a500a22a (127.0.97.1:41333), 63969b25b0244237b7729b899aea297c (127.0.97.2:44427)
I20260929 20:46:41.827616 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c674d49e75f24bd7b88c9808ddb764fb" candidate_uuid: "07b852f019974bbf97d358e63323f43e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de2c783b8c4c401ba731af02a500a22a"
I20260929 20:46:41.827739 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c674d49e75f24bd7b88c9808ddb764fb" candidate_uuid: "07b852f019974bbf97d358e63323f43e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63969b25b0244237b7729b899aea297c"
I20260929 20:46:41.827829 1671 raft_consensus.cc:3037] T c674d49e75f24bd7b88c9808ddb764fb P de2c783b8c4c401ba731af02a500a22a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:41.827962 1806 raft_consensus.cc:3037] T c674d49e75f24bd7b88c9808ddb764fb P 63969b25b0244237b7729b899aea297c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:41.830629 1806 raft_consensus.cc:2454] T c674d49e75f24bd7b88c9808ddb764fb P 63969b25b0244237b7729b899aea297c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b852f019974bbf97d358e63323f43e in term 1.
I20260929 20:46:41.830695 1671 raft_consensus.cc:2454] T c674d49e75f24bd7b88c9808ddb764fb P de2c783b8c4c401ba731af02a500a22a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b852f019974bbf97d358e63323f43e in term 1.
I20260929 20:46:41.831064 1877 leader_election.cc:302] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b852f019974bbf97d358e63323f43e, de2c783b8c4c401ba731af02a500a22a; no voters:
I20260929 20:46:41.831328 2021 raft_consensus.cc:2781] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:46:41.831738 2021 raft_consensus.cc:686] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e [term 1 LEADER]: Becoming Leader. State: Replica: 07b852f019974bbf97d358e63323f43e, State: Running, Role: LEADER
I20260929 20:46:41.832118 2021 consensus_queue.cc:237] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } }
I20260929 20:46:41.833911 2057 tablet_replica.cc:439] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e: TxnStatusTablet state changed. Reason: New leader 07b852f019974bbf97d358e63323f43e. Latest consensus state: current_term: 1 leader_uuid: "07b852f019974bbf97d358e63323f43e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } } }
I20260929 20:46:41.835338 2057 tablet_replica.cc:442] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:46:41.835664 1508 catalog_manager.cc:5495] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e reported cstate change: term changed from 0 to 1, leader changed from <none> to 07b852f019974bbf97d358e63323f43e (127.0.97.3). New cstate: current_term: 1 leader_uuid: "07b852f019974bbf97d358e63323f43e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } health_report { overall_health: HEALTHY } } }
I20260929 20:46:41.835916 2059 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 20:46:41.836090 2059 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 20:46:41.864115 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.857824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39468 (local address 127.0.97.3:42149)
0929 20:46:41.858404 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:41.858410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:41.858432 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:46:41.858521 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:41.858526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:41.858614 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:41.858732 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:46:41.859137 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.859969 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.861843 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.862328 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.862416 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.862958 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:46:41.863572 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:46:41.863672 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:41.863807 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 20:46:41.863863 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":326,"threads_started":1}
I20260929 20:46:41.864229 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.857727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42149 (local address 127.0.0.1:39468)
0929 20:46:41.857961 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:41.857981 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:41.858129 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:41.858783 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:41.858788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:41.858801 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:46:41.858977 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.858988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.860102 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.860108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.861709 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.861723 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.862469 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.862476 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.862676 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.862746 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:46:41.863677 (+ 931us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:46:41.863698 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 20:46:41.863790 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 20:46:41.863936 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 20:46:41.868407 1806 raft_consensus.cc:1261] T c674d49e75f24bd7b88c9808ddb764fb P 63969b25b0244237b7729b899aea297c [term 1 FOLLOWER]: Refusing update from remote peer 07b852f019974bbf97d358e63323f43e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:46:41.868603 1671 raft_consensus.cc:1261] T c674d49e75f24bd7b88c9808ddb764fb P de2c783b8c4c401ba731af02a500a22a [term 1 FOLLOWER]: Refusing update from remote peer 07b852f019974bbf97d358e63323f43e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:46:41.869084 2057 consensus_queue.cc:1035] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e [LEADER]: Connected to new peer: Peer: permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:46:41.869432 2021 consensus_queue.cc:1035] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:46:41.879998 2020 tablet_replica.cc:439] T c674d49e75f24bd7b88c9808ddb764fb P 63969b25b0244237b7729b899aea297c: TxnStatusTablet state changed. Reason: New leader 07b852f019974bbf97d358e63323f43e. Latest consensus state: current_term: 1 leader_uuid: "07b852f019974bbf97d358e63323f43e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } } }
I20260929 20:46:41.880286 2020 tablet_replica.cc:442] T c674d49e75f24bd7b88c9808ddb764fb P 63969b25b0244237b7729b899aea297c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:46:41.882759 2063 mvcc.cc:205] Tried to move back new op lower bound from 7334767828445941760 to 7334767828307681280. Current Snapshot: MvccSnapshot[applied={T|T < 7334767828445941760}]
I20260929 20:46:41.884562 2033 tablet_replica.cc:439] T c674d49e75f24bd7b88c9808ddb764fb P de2c783b8c4c401ba731af02a500a22a: TxnStatusTablet state changed. Reason: New leader 07b852f019974bbf97d358e63323f43e. Latest consensus state: current_term: 1 leader_uuid: "07b852f019974bbf97d358e63323f43e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } } }
I20260929 20:46:41.884833 2033 tablet_replica.cc:442] T c674d49e75f24bd7b88c9808ddb764fb P de2c783b8c4c401ba731af02a500a22a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:46:41.887281 2057 tablet_replica.cc:439] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07b852f019974bbf97d358e63323f43e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } } }
I20260929 20:46:41.887509 2057 tablet_replica.cc:442] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:46:41.890040 2021 tablet_replica.cc:439] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07b852f019974bbf97d358e63323f43e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } } }
I20260929 20:46:41.890327 2021 tablet_replica.cc:442] T c674d49e75f24bd7b88c9808ddb764fb P 07b852f019974bbf97d358e63323f43e: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:46:41.890914 2033 tablet_replica.cc:439] T c674d49e75f24bd7b88c9808ddb764fb P de2c783b8c4c401ba731af02a500a22a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07b852f019974bbf97d358e63323f43e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } } }
I20260929 20:46:41.891685 2033 tablet_replica.cc:442] T c674d49e75f24bd7b88c9808ddb764fb P de2c783b8c4c401ba731af02a500a22a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:46:41.892840 2020 tablet_replica.cc:439] T c674d49e75f24bd7b88c9808ddb764fb P 63969b25b0244237b7729b899aea297c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07b852f019974bbf97d358e63323f43e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de2c783b8c4c401ba731af02a500a22a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41333 } } peers { permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 } } peers { permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 } } }
I20260929 20:46:41.893085 2020 tablet_replica.cc:442] T c674d49e75f24bd7b88c9808ddb764fb P 63969b25b0244237b7729b899aea297c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:46:41.909690 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.904613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38187 (local address 127.0.97.3:42149)
0929 20:46:41.904758 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:41.904762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:41.905337 (+ 575us) server_negotiation.cc:407] Connection header received
0929 20:46:41.905519 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:41.905524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:41.905590 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:41.905662 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:46:41.906075 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.906826 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.908104 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.908525 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.908631 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.908993 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:46:41.909123 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:46:41.909196 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:41.909325 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 20:46:41.909396 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 20:46:41.909742 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.904658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42149 (local address 127.0.97.1:38187)
0929 20:46:41.905247 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:41.905262 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:41.905406 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:41.905683 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:41.905687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:41.905698 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:46:41.905936 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.905951 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.906939 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.906952 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.907979 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.907990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.908659 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.908664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.908793 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.908843 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:46:41.909209 (+ 366us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:46:41.909220 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 20:46:41.909288 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 20:46:41.909410 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":344,"threads_started":1}
I20260929 20:46:41.917454 1804 raft_consensus.cc:1261] T c39795da2f784bf592b69cd890fa0301 P 63969b25b0244237b7729b899aea297c [term 1 FOLLOWER]: Refusing update from remote peer de2c783b8c4c401ba731af02a500a22a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:46:41.917464 1948 raft_consensus.cc:1261] T c39795da2f784bf592b69cd890fa0301 P 07b852f019974bbf97d358e63323f43e [term 1 FOLLOWER]: Refusing update from remote peer de2c783b8c4c401ba731af02a500a22a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:46:41.918006 2033 consensus_queue.cc:1035] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b852f019974bbf97d358e63323f43e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:46:41.918303 2019 consensus_queue.cc:1035] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a [LEADER]: Connected to new peer: Peer: permanent_uuid: "63969b25b0244237b7729b899aea297c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:46:41.927991 2065 mvcc.cc:205] Tried to move back new op lower bound from 7334767828649455616 to 7334767827650801664. Current Snapshot: MvccSnapshot[applied={T|T < 7334767828649455616}]
I20260929 20:46:41.953102 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.946487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60135 (local address 127.0.97.1:41333)
0929 20:46:41.946644 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:41.946650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:41.947300 (+ 650us) server_negotiation.cc:407] Connection header received
0929 20:46:41.947470 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:41.947473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:41.947528 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:41.947594 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:46:41.948161 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.948988 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.951278 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.951915 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.952047 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.952431 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:46:41.952553 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:46:41.952639 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:41.952754 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 20:46:41.952850 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 20:46:41.953269 2078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:41.946533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41333 (local address 127.0.97.3:60135)
0929 20:46:41.947210 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:41.947229 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:41.947367 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:41.947624 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:41.947628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:41.947641 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:46:41.947998 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.948009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.949122 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.949126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.951155 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:41.951170 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:41.952067 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:41.952071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:41.952206 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:41.952269 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:46:41.952641 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:46:41.952649 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 20:46:41.952715 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 20:46:41.952864 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":276,"threads_started":1}
I20260929 20:46:41.987836 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:46:41.988103 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:46:41.990968 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 054502fd0ff74352bbe80ef9027f0476: Sending DeleteTablet for 3 replicas of tablet c39795da2f784bf592b69cd890fa0301
I20260929 20:46:41.991812 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet c39795da2f784bf592b69cd890fa0301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:46:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57822
I20260929 20:46:41.991824 1923 tablet_service.cc:1514] Processing DeleteTablet for tablet c39795da2f784bf592b69cd890fa0301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:46:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39452
I20260929 20:46:41.992225 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet c39795da2f784bf592b69cd890fa0301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:46:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51496
I20260929 20:46:42.006755 2081 tablet_replica.cc:330] T c39795da2f784bf592b69cd890fa0301 P 07b852f019974bbf97d358e63323f43e: stopping tablet replica
I20260929 20:46:42.006994 2081 raft_consensus.cc:2229] T c39795da2f784bf592b69cd890fa0301 P 07b852f019974bbf97d358e63323f43e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:46:42.007315 2081 raft_consensus.cc:2258] T c39795da2f784bf592b69cd890fa0301 P 07b852f019974bbf97d358e63323f43e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:46:42.007762 2083 tablet_replica.cc:330] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a: stopping tablet replica
I20260929 20:46:42.008023 2083 raft_consensus.cc:2229] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:46:42.008869 2081 ts_tablet_manager.cc:1905] T c39795da2f784bf592b69cd890fa0301 P 07b852f019974bbf97d358e63323f43e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:46:42.009958 2083 raft_consensus.cc:2258] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:46:42.011008 2083 ts_tablet_manager.cc:1905] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:46:42.011547 2082 tablet_replica.cc:330] T c39795da2f784bf592b69cd890fa0301 P 63969b25b0244237b7729b899aea297c: stopping tablet replica
I20260929 20:46:42.015208 2082 raft_consensus.cc:2229] T c39795da2f784bf592b69cd890fa0301 P 63969b25b0244237b7729b899aea297c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:46:42.015566 2082 raft_consensus.cc:2258] T c39795da2f784bf592b69cd890fa0301 P 63969b25b0244237b7729b899aea297c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:46:42.016340 2081 ts_tablet_manager.cc:1918] T c39795da2f784bf592b69cd890fa0301 P 07b852f019974bbf97d358e63323f43e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:46:42.016481 2081 log.cc:1196] T c39795da2f784bf592b69cd890fa0301 P 07b852f019974bbf97d358e63323f43e: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/c39795da2f784bf592b69cd890fa0301
I20260929 20:46:42.016770 2082 ts_tablet_manager.cc:1905] T c39795da2f784bf592b69cd890fa0301 P 63969b25b0244237b7729b899aea297c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:46:42.016934 2081 ts_tablet_manager.cc:1939] T c39795da2f784bf592b69cd890fa0301 P 07b852f019974bbf97d358e63323f43e: Deleting consensus metadata
I20260929 20:46:42.017985 1495 catalog_manager.cc:4841] TS 07b852f019974bbf97d358e63323f43e (127.0.97.3:42149): tablet c39795da2f784bf592b69cd890fa0301 (table test-table [id=91f5599254d344608509decb3e87392e]) successfully deleted
I20260929 20:46:42.021111 2083 ts_tablet_manager.cc:1918] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:46:42.021221 2083 log.cc:1196] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/c39795da2f784bf592b69cd890fa0301
I20260929 20:46:42.021637 2083 ts_tablet_manager.cc:1939] T c39795da2f784bf592b69cd890fa0301 P de2c783b8c4c401ba731af02a500a22a: Deleting consensus metadata
I20260929 20:46:42.022650 1496 catalog_manager.cc:4841] TS de2c783b8c4c401ba731af02a500a22a (127.0.97.1:41333): tablet c39795da2f784bf592b69cd890fa0301 (table test-table [id=91f5599254d344608509decb3e87392e]) successfully deleted
I20260929 20:46:42.024044 2082 ts_tablet_manager.cc:1918] T c39795da2f784bf592b69cd890fa0301 P 63969b25b0244237b7729b899aea297c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:46:42.024171 2082 log.cc:1196] T c39795da2f784bf592b69cd890fa0301 P 63969b25b0244237b7729b899aea297c: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/c39795da2f784bf592b69cd890fa0301
Sep 29 20:46:42 dist-test-slave-blv7 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714801, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:46:42.024582 2082 ts_tablet_manager.cc:1939] T c39795da2f784bf592b69cd890fa0301 P 63969b25b0244237b7729b899aea297c: Deleting consensus metadata
I20260929 20:46:42.025511 1495 catalog_manager.cc:4841] TS 63969b25b0244237b7729b899aea297c (127.0.97.2:44427): tablet c39795da2f784bf592b69cd890fa0301 (table test-table [id=91f5599254d344608509decb3e87392e]) successfully deleted
I20260929 20:46:42.030308 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:42.009048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51508 (local address 127.0.97.1:41333)
0929 20:46:42.009230 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:42.009236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:42.012267 (+ 3031us) server_negotiation.cc:407] Connection header received
0929 20:46:42.012663 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:42.012669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:42.012753 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:42.012825 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:42.014728 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:42.015883 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:42.017636 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:42.018219 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:42.018308 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:42.025140 (+ 6832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:42.025151 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:42.025166 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:42.025202 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:42.027887 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:42.028500 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:42.028507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:42.028510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:42.028622 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:42.028940 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:42.028945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:42.028948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:42.029282 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:42.029559 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:42.029809 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 20:46:42.030016 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 20:46:42.030972 1711 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51508
I20260929 20:46:42.046260 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:42.033477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47530 (local address 127.0.97.62:45269)
0929 20:46:42.033684 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:42.033689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:42.033770 (+ 81us) server_negotiation.cc:407] Connection header received
0929 20:46:42.033963 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:42.033969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:42.034026 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:42.034102 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:42.035699 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:42.036406 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:42.038130 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:42.038547 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:42.038636 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:42.041441 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:42.041454 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:42.041458 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:42.041489 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:42.043953 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:42.044670 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:42.044674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:42.044676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:42.044780 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:42.045112 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:42.045115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:42.045117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:42.045383 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:42.045568 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:42.045884 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 20:46:42.046041 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260929 20:46:42.046999 1510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47530
I20260929 20:46:42.048748 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 1581
I20260929 20:46:42.060648 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:42.049550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51516 (local address 127.0.97.1:41333)
0929 20:46:42.049726 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:42.049732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:42.050062 (+ 330us) server_negotiation.cc:407] Connection header received
0929 20:46:42.050230 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:42.050236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:42.050328 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:42.050408 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:42.051598 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:42.052283 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:42.053384 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:42.053789 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:42.053889 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:42.055974 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:42.055979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:42.055982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:42.056012 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:42.058331 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:42.058893 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:42.058898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:42.058902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:42.059058 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:42.059366 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:42.059371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:42.059374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:42.059624 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:42.059832 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:42.060115 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 20:46:42.060304 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 20:46:42.061163 1711 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51516
W20260929 20:46:42.061668 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 1581: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:46:42.061849 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 1581
I20260929 20:46:42.079262 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 1720
Sep 29 20:46:42 dist-test-slave-blv7 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714801, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:46:42.093031 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:42.079935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57830 (local address 127.0.97.2:44427)
0929 20:46:42.080093 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:42.080099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:42.080175 (+ 76us) server_negotiation.cc:407] Connection header received
0929 20:46:42.080369 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:42.080373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:42.080452 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:42.080546 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:42.081681 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:42.082341 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:42.083597 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:42.083996 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:42.084101 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:42.087913 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:42.087925 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:42.087939 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:42.087975 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:42.090469 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:42.091046 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:42.091054 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:42.091057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:42.091204 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:42.091613 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:42.091618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:42.091621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:42.091936 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:42.092211 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:42.092487 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 20:46:42.092704 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 20:46:42.093575 1850 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57830
W20260929 20:46:42.094123 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 1720: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:46:42.094264 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 1720
I20260929 20:46:42.113471 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 1859
Sep 29 20:46:42 dist-test-slave-blv7 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714801, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:46:42.132468 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:42.114210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39474 (local address 127.0.97.3:42149)
0929 20:46:42.114380 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:42.114386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:42.114517 (+ 131us) server_negotiation.cc:407] Connection header received
0929 20:46:42.114705 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:42.114711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:42.114785 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:42.114894 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:42.116456 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:42.117632 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:42.119413 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:42.120151 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:42.120269 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:42.125364 (+ 5095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:42.125376 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:42.125393 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:42.125430 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:42.129380 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:42.130136 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:42.130144 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:42.130148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:42.130299 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:42.130656 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:42.130661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:42.130668 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:42.131117 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:42.131437 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:42.131690 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 20:46:42.131931 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 20:46:42.133215 1989 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39474
W20260929 20:46:42.133899 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 1859: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:46:42.134056 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 1859
I20260929 20:46:42.149185 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 1478
I20260929 20:46:42.150149 1550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:46:42.259223 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 1478
2026-09-29T20:46:42Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4191 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:46:42 dist-test-slave-blv7 krb5kdc[2092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:46:42 dist-test-slave-blv7 krb5kdc[2092](info): set up 2 sockets
Sep 29 20:46:42 dist-test-slave-blv7 krb5kdc[2092](info): commencing operation
krb5kdc: starting...
W20260929 20:46:43.254917 388 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:46:43 dist-test-slave-blv7 krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:46:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:46:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:46:43.418864 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36609
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:43.726315 2105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:43.726604 2105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:43.726745 2105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:43.733934 2105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:46:43.734035 2105 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 20:46:43.734107 2105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:43.734189 2105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:43.734265 2105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:46:43.734344 2105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:46:43.744127 2105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36609
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:43.745302 2105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:43.746618 2105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:43.758244 2110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:43.758188 2113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:43.758217 2111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:43.758576 2105 server_base.cc:1034] running on GCE node
I20260929 20:46:43.759562 2105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:43.760811 2105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:43.762032 2105 hybrid_clock.cc:648] HybridClock initialized: now 1790714803761967 us; error 41 us; skew 500 ppm
I20260929 20:46:43.762344 2105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:43 dist-test-slave-blv7 krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:43.765240 2105 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:46:43.766606 2105 webserver.cc:460] Webserver started at http://127.0.97.62:39399/ using document root <none> and password file <none>
I20260929 20:46:43.767313 2105 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:43.767407 2105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:43.767688 2105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:43.770448 2105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "4df9c243e8cc423da9266e8ec6d17052"
format_stamp: "Formatted at 2026-09-29 20:46:43 on dist-test-slave-blv7"
server_key: "61be0c0c2b465ca0b96d897202464c35"
server_key_iv: "847352352b5eda9de4b8dc0e0f5732bd"
server_key_version: "encryptionkey@0"
I20260929 20:46:43.771207 2105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "4df9c243e8cc423da9266e8ec6d17052"
format_stamp: "Formatted at 2026-09-29 20:46:43 on dist-test-slave-blv7"
server_key: "61be0c0c2b465ca0b96d897202464c35"
server_key_iv: "847352352b5eda9de4b8dc0e0f5732bd"
server_key_version: "encryptionkey@0"
I20260929 20:46:43.775935 2105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:46:43.779124 2120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:43.780313 2105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:46:43.780449 2105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "4df9c243e8cc423da9266e8ec6d17052"
format_stamp: "Formatted at 2026-09-29 20:46:43 on dist-test-slave-blv7"
server_key: "61be0c0c2b465ca0b96d897202464c35"
server_key_iv: "847352352b5eda9de4b8dc0e0f5732bd"
server_key_version: "encryptionkey@0"
I20260929 20:46:43.780651 2105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:43.811561 2105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:43.815448 2105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:43.815671 2105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:43.834575 2105 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36609
I20260929 20:46:43.834601 2181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36609 every 8 connection(s)
I20260929 20:46:43.835961 2105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:46:43.839475 2182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:43.843602 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 2105
I20260929 20:46:43.843786 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:46:43.844517 388 external_mini_cluster.cc:1428] Setting key 4b942626016c768a9347a358286c661f
I20260929 20:46:43.850337 2182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052: Bootstrap starting.
I20260929 20:46:43.853653 2182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:43.854710 2182 log.cc:824] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052: Log is configured to *not* fsync() on all Append() calls
I20260929 20:46:43.857705 2182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052: No bootstrap required, opened a new log
Sep 29 20:46:43 dist-test-slave-blv7 krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:43.865955 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:43.846448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42232 (local address 127.0.97.62:36609)
0929 20:46:43.847191 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:43.847201 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:43.847244 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:46:43.849041 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:43.849062 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:43.849428 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:43.849731 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:43.851321 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:43.852611 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:43.854198 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:43.854914 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:43.855091 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:43.859472 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:43.859482 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:43.859498 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:43.859529 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:43.862288 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:43.862991 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:43.862997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:43.863002 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:43.863196 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:43.863546 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:43.863549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:43.863552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:43.864011 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:43.864214 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:43.864613 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 20:46:43.864946 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":331,"threads_started":1}
I20260929 20:46:43.865922 2182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df9c243e8cc423da9266e8ec6d17052" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36609 } }
I20260929 20:46:43.866276 2182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:43.866381 2182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df9c243e8cc423da9266e8ec6d17052, State: Initialized, Role: FOLLOWER
I20260929 20:46:43.866935 2182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df9c243e8cc423da9266e8ec6d17052" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36609 } }
I20260929 20:46:43.867199 2182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:46:43.867359 2182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:46:43.867532 2182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:43.869570 2182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df9c243e8cc423da9266e8ec6d17052" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36609 } }
I20260929 20:46:43.869987 2182 leader_election.cc:302] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4df9c243e8cc423da9266e8ec6d17052; no voters:
I20260929 20:46:43.870455 2182 leader_election.cc:288] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:46:43.870570 2187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:46:43.871186 2187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052 [term 1 LEADER]: Becoming Leader. State: Replica: 4df9c243e8cc423da9266e8ec6d17052, State: Running, Role: LEADER
I20260929 20:46:43.871656 2187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df9c243e8cc423da9266e8ec6d17052" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36609 } }
I20260929 20:46:43.872260 2182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:46:43.874104 2188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4df9c243e8cc423da9266e8ec6d17052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df9c243e8cc423da9266e8ec6d17052" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36609 } } }
I20260929 20:46:43.874571 2189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4df9c243e8cc423da9266e8ec6d17052. Latest consensus state: current_term: 1 leader_uuid: "4df9c243e8cc423da9266e8ec6d17052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df9c243e8cc423da9266e8ec6d17052" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36609 } } }
I20260929 20:46:43.875298 2189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052 [sys.catalog]: This master's current role is: LEADER
I20260929 20:46:43.876173 2188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052 [sys.catalog]: This master's current role is: LEADER
I20260929 20:46:43.876299 2194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:46:43.881479 2194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:46:43.889654 2194 catalog_manager.cc:1347] Generated new cluster ID: 8989e46ff7cf435a8db03e4c920e3c84
I20260929 20:46:43.889758 2194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:46:43.910466 2194 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:46:43.911811 2194 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:46:43.920259 2194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052: Generated new TSK 0
I20260929 20:46:43.921056 2194 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:46:43.973990 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36609
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:44.295250 2208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:44.295517 2208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:44.295667 2208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:44.302899 2208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:44.303004 2208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:44.303154 2208 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 20:46:44.303215 2208 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:46:44.303313 2208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:46:44.312988 2208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:36609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:44.314213 2208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:44.315537 2208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:44.328071 2214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:44.328042 2216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:44.328233 2213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:44.328936 2208 server_base.cc:1034] running on GCE node
I20260929 20:46:44.329506 2208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:44.330431 2208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:44.331619 2208 hybrid_clock.cc:648] HybridClock initialized: now 1790714804331575 us; error 34 us; skew 500 ppm
I20260929 20:46:44.331895 2208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:44 dist-test-slave-blv7 krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:44.334759 2208 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:46:44.336148 2208 webserver.cc:460] Webserver started at http://127.0.97.1:44881/ using document root <none> and password file <none>
I20260929 20:46:44.336787 2208 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:44.336898 2208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:44.337169 2208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:44.339998 2208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "3ed8101318574e84b35484a916fa5515"
format_stamp: "Formatted at 2026-09-29 20:46:44 on dist-test-slave-blv7"
server_key: "d603d4a5dab864045b8f480a038cfdf6"
server_key_iv: "98aa77039a38e29daa0065cd9e653ec0"
server_key_version: "encryptionkey@0"
I20260929 20:46:44.340739 2208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3ed8101318574e84b35484a916fa5515"
format_stamp: "Formatted at 2026-09-29 20:46:44 on dist-test-slave-blv7"
server_key: "d603d4a5dab864045b8f480a038cfdf6"
server_key_iv: "98aa77039a38e29daa0065cd9e653ec0"
server_key_version: "encryptionkey@0"
I20260929 20:46:44.345506 2208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:46:44.348682 2223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:44.349952 2208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:46:44.350088 2208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "3ed8101318574e84b35484a916fa5515"
format_stamp: "Formatted at 2026-09-29 20:46:44 on dist-test-slave-blv7"
server_key: "d603d4a5dab864045b8f480a038cfdf6"
server_key_iv: "98aa77039a38e29daa0065cd9e653ec0"
server_key_version: "encryptionkey@0"
I20260929 20:46:44.350260 2208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:44.366803 2208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:44.370841 2208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:44.371133 2208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:44.373824 2208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:46:44.373931 2208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:44.374040 2208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:46:44.374130 2208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:46:44 dist-test-slave-blv7 krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:44.397554 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:44.374975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36609 (local address 127.0.97.1:53205)
0929 20:46:44.375764 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:44.375802 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:44.377700 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:44.378477 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:44.378495 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:44.379325 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:44.380320 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:44.380354 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:44.381597 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:44.381603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:44.383888 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:44.383910 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:44.384999 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:44.385007 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:44.385330 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:44.386040 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:44.386078 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:44.388878 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:44.392680 (+ 3802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:44.392688 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:44.392708 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:44.393247 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:44.393672 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:44.393677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:44.393681 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:44.393954 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:44.394593 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:44.394599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:44.395120 (+ 521us) client_negotiation.cc:769] Sending connection context
0929 20:46:44.395475 (+ 355us) client_negotiation.cc:240] Negotiation successful
0929 20:46:44.396030 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":321,"threads_started":1}
I20260929 20:46:44.398535 2232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:44.374313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53205 (local address 127.0.97.62:36609)
0929 20:46:44.375045 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:44.375051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:44.375908 (+ 857us) server_negotiation.cc:407] Connection header received
0929 20:46:44.378044 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:44.378048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:44.378111 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:44.378188 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:44.380533 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:44.381426 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:44.384102 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:44.384751 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:44.384880 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:44.389123 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:44.389130 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:44.389136 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:44.389181 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:44.392541 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:44.393386 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:44.393393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:44.393396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:44.393554 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:44.394069 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:44.394074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:44.394078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:44.394402 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:44.394533 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:44.398107 (+ 3574us) server_negotiation.cc:299] Negotiation successful
0929 20:46:44.398302 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":513,"threads_started":1}
I20260929 20:46:44.415512 2232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:44.403277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42067 (local address 127.0.97.62:36609)
0929 20:46:44.403447 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:44.403452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:44.404205 (+ 753us) server_negotiation.cc:407] Connection header received
0929 20:46:44.404342 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:44.404348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:44.404414 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:44.404486 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:44.405290 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:44.405980 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:44.407242 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:44.407817 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:44.407944 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:44.410225 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:44.410231 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:44.410236 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:44.410271 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:44.412968 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:44.413777 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:44.413782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:44.413786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:44.413902 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:44.414207 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:44.414212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:44.414216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:44.414484 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:44.414604 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:44.414966 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 20:46:44.415182 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:46:44.415660 2302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:44.403490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36609 (local address 127.0.97.1:42067)
0929 20:46:44.404097 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:44.404116 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:44.404239 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:44.404520 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:44.404524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:44.404918 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:44.405169 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:44.405177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:44.406111 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:44.406115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:44.407120 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:44.407131 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:44.408168 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:44.408172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:44.408316 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:44.409006 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:44.409026 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:44.410049 (+ 1023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:44.413087 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:44.413092 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:44.413096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:44.413675 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:44.413995 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:44.414000 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:44.414003 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:44.414118 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:44.414587 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:44.414591 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:44.414752 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 20:46:44.414865 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 20:46:44.415208 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":346,"threads_started":1}
I20260929 20:46:44.427057 2208 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44991
I20260929 20:46:44.427095 2343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44991 every 8 connection(s)
I20260929 20:46:44.428201 2208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:46:44.430027 2344 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36609
I20260929 20:46:44.430214 2344 heartbeater.cc:464] Registering TS with master...
I20260929 20:46:44.430780 2344 heartbeater.cc:511] Master 127.0.97.62:36609 requested a full tablet report, sending...
I20260929 20:46:44.432559 2137 ts_manager.cc:194] Registered new tserver with Master: 3ed8101318574e84b35484a916fa5515 (127.0.97.1:44991)
I20260929 20:46:44.432653 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 2208
I20260929 20:46:44.432814 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:46:44.433383 388 external_mini_cluster.cc:1428] Setting key fc29fe8ff0924e2e71a5622029a6d7dc
I20260929 20:46:44.434099 2137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53205
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:46:44.469780 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36609
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:44.766326 2348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:44.766559 2348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:44.766683 2348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:44.773931 2348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:44.774024 2348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:44.774142 2348 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 20:46:44.774197 2348 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:46:44.774271 2348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:46:44.783699 2348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:36609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:44.784924 2348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:44.786134 2348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:44.798943 2354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:44.798919 2356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:44.798914 2353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:44.799710 2348 server_base.cc:1034] running on GCE node
I20260929 20:46:44.800261 2348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:44.801223 2348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:44.802435 2348 hybrid_clock.cc:648] HybridClock initialized: now 1790714804802375 us; error 51 us; skew 500 ppm
I20260929 20:46:44.802726 2348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:44 dist-test-slave-blv7 krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:44.808832 2348 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:46:44.810247 2348 webserver.cc:460] Webserver started at http://127.0.97.2:45039/ using document root <none> and password file <none>
I20260929 20:46:44.810866 2348 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:44.810972 2348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:44.811275 2348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:44.813980 2348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "07c034afa3e44dcdaf52f96e4c4f14dc"
format_stamp: "Formatted at 2026-09-29 20:46:44 on dist-test-slave-blv7"
server_key: "26339a08822cf7ba1cc5aa0d84cf79cb"
server_key_iv: "6f5ab2e98251c4650b59d067e2c27f05"
server_key_version: "encryptionkey@0"
I20260929 20:46:44.814706 2348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "07c034afa3e44dcdaf52f96e4c4f14dc"
format_stamp: "Formatted at 2026-09-29 20:46:44 on dist-test-slave-blv7"
server_key: "26339a08822cf7ba1cc5aa0d84cf79cb"
server_key_iv: "6f5ab2e98251c4650b59d067e2c27f05"
server_key_version: "encryptionkey@0"
I20260929 20:46:44.819535 2348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260929 20:46:44.822747 2363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:44.824013 2348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 20:46:44.824154 2348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "07c034afa3e44dcdaf52f96e4c4f14dc"
format_stamp: "Formatted at 2026-09-29 20:46:44 on dist-test-slave-blv7"
server_key: "26339a08822cf7ba1cc5aa0d84cf79cb"
server_key_iv: "6f5ab2e98251c4650b59d067e2c27f05"
server_key_version: "encryptionkey@0"
I20260929 20:46:44.824319 2348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:44.837572 2348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:44.841652 2348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:44.841945 2348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:44.844694 2348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:46:44.844828 2348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:44.844949 2348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:46:44.845039 2348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:46:44 dist-test-slave-blv7 krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:44.865311 2232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:44.845252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50403 (local address 127.0.97.62:36609)
0929 20:46:44.845425 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:44.845431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:44.846996 (+ 1565us) server_negotiation.cc:407] Connection header received
0929 20:46:44.849537 (+ 2541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:44.849542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:44.849622 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:44.849707 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:44.851598 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:44.852588 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:44.854407 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:44.854818 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:44.854926 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:44.859107 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:44.859114 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:44.859119 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:44.859149 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:44.861958 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:44.862850 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:44.862857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:44.862860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:44.863048 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:44.863567 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:44.863572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:44.863576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:44.863922 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:44.864099 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:44.864779 (+ 680us) server_negotiation.cc:299] Negotiation successful
0929 20:46:44.864997 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:46:44.866261 2372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:44.845977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36609 (local address 127.0.97.2:50403)
0929 20:46:44.846843 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:44.846884 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:44.849281 (+ 2397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:44.849958 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:44.849981 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:44.850672 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:44.851439 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:44.851457 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:44.852726 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:44.852730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:44.854265 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:44.854277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:44.855054 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:44.855063 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:44.855269 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:44.856053 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:44.856086 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:44.858800 (+ 2714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:44.862088 (+ 3288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:44.862097 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:44.862114 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:44.862688 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:44.863176 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:44.863181 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:44.863185 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:44.863441 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:44.864092 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:44.864099 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:44.864418 (+ 319us) client_negotiation.cc:769] Sending connection context
0929 20:46:44.864652 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 20:46:44.865121 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":379,"threads_started":1}
I20260929 20:46:44.883872 2425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:44.871448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36609 (local address 127.0.97.2:37343)
0929 20:46:44.871997 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:44.872018 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:44.872131 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:44.872416 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:44.872420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:44.872832 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:44.873092 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:44.873101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:44.874082 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:44.874087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:44.875148 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:44.875163 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:44.876218 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:44.876222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:44.876372 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:44.876983 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:44.876998 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:44.877636 (+ 638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:44.881040 (+ 3404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:44.881044 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:44.881046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:44.881550 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:44.881917 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:44.881921 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:44.881928 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:44.882076 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:44.882691 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:44.882695 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:44.882966 (+ 271us) client_negotiation.cc:769] Sending connection context
0929 20:46:44.883194 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 20:46:44.883516 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":314,"threads_started":1}
I20260929 20:46:44.883991 2232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:44.871291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37343 (local address 127.0.97.62:36609)
0929 20:46:44.871445 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:44.871450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:44.872093 (+ 643us) server_negotiation.cc:407] Connection header received
0929 20:46:44.872225 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:44.872230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:44.872314 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:44.872393 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:44.873210 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:44.873964 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:44.875315 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:44.876048 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:44.876197 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:44.877814 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:44.877823 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:44.877829 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:44.877872 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:44.880903 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:44.881649 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:44.881654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:44.881658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:44.881785 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:44.882192 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:44.882197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:44.882201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:44.882585 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:44.882706 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:44.883317 (+ 611us) server_negotiation.cc:299] Negotiation successful
0929 20:46:44.883567 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:46:44.898605 2348 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45057
I20260929 20:46:44.898638 2482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45057 every 8 connection(s)
I20260929 20:46:44.899840 2348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:46:44.901819 2483 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36609
I20260929 20:46:44.902027 2483 heartbeater.cc:464] Registering TS with master...
I20260929 20:46:44.902582 2483 heartbeater.cc:511] Master 127.0.97.62:36609 requested a full tablet report, sending...
I20260929 20:46:44.903721 2136 ts_manager.cc:194] Registered new tserver with Master: 07c034afa3e44dcdaf52f96e4c4f14dc (127.0.97.2:45057)
I20260929 20:46:44.904552 2136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50403
I20260929 20:46:44.904728 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 2348
I20260929 20:46:44.904872 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:46:44.905287 388 external_mini_cluster.cc:1428] Setting key 0c19b022a806dd9036ef8027aee553e1
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:46:44.937985 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36609
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:45.234748 2487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:45.235010 2487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:45.235152 2487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:45.242174 2487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:45.242270 2487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:45.242359 2487 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 20:46:45.242416 2487 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:46:45.242482 2487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:46:45.251725 2487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:36609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:45.252843 2487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:45.254118 2487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:45.265923 2492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:45.265944 2493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:45.265877 2495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:45.266733 2487 server_base.cc:1034] running on GCE node
I20260929 20:46:45.267285 2487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:45.268229 2487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:45.269416 2487 hybrid_clock.cc:648] HybridClock initialized: now 1790714805269379 us; error 53 us; skew 500 ppm
I20260929 20:46:45.269692 2487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:45 dist-test-slave-blv7 krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:45.272662 2487 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:46:45.273962 2487 webserver.cc:460] Webserver started at http://127.0.97.3:36859/ using document root <none> and password file <none>
I20260929 20:46:45.274559 2487 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:45.274667 2487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:45.274937 2487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:45.277668 2487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "ac1a30935b2647609e7a3b3eb055b6ad"
format_stamp: "Formatted at 2026-09-29 20:46:45 on dist-test-slave-blv7"
server_key: "5694e6e3b0c88da8e5f3e311249b8818"
server_key_iv: "a1898da39d0aea8317f1589736e36a6f"
server_key_version: "encryptionkey@0"
I20260929 20:46:45.278399 2487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ac1a30935b2647609e7a3b3eb055b6ad"
format_stamp: "Formatted at 2026-09-29 20:46:45 on dist-test-slave-blv7"
server_key: "5694e6e3b0c88da8e5f3e311249b8818"
server_key_iv: "a1898da39d0aea8317f1589736e36a6f"
server_key_version: "encryptionkey@0"
I20260929 20:46:45.283154 2487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260929 20:46:45.286413 2502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:45.287684 2487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:46:45.287823 2487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "ac1a30935b2647609e7a3b3eb055b6ad"
format_stamp: "Formatted at 2026-09-29 20:46:45 on dist-test-slave-blv7"
server_key: "5694e6e3b0c88da8e5f3e311249b8818"
server_key_iv: "a1898da39d0aea8317f1589736e36a6f"
server_key_version: "encryptionkey@0"
I20260929 20:46:45.288025 2487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:45.310659 2487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:45.314306 2487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:45.314558 2487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:45.317005 2487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:46:45.317121 2487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:45.317226 2487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:46:45.317302 2487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:46:45 dist-test-slave-blv7 krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:45.337523 2232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.317652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44591 (local address 127.0.97.62:36609)
0929 20:46:45.317808 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:45.317814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:45.319239 (+ 1425us) server_negotiation.cc:407] Connection header received
0929 20:46:45.321466 (+ 2227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:45.321472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:45.321555 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:45.321644 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:45.323789 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.324811 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.327195 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.327752 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.327877 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.332253 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:45.332259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:45.332262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:45.332291 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:45.334654 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:45.335394 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:45.335398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:45.335400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:45.335536 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:45.335912 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:45.335916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:45.335918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:45.336158 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:45.336279 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:45.336959 (+ 680us) server_negotiation.cc:299] Negotiation successful
0929 20:46:45.337169 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 20:46:45.338788 2511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.318311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36609 (local address 127.0.97.3:44591)
0929 20:46:45.319105 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:45.319145 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:45.321211 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:45.322072 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:45.322086 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:45.322742 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:45.323610 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.323629 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.324948 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.324957 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.327064 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.327078 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.327987 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.327996 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.328297 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.329100 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:45.329132 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:45.332050 (+ 2918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:45.334778 (+ 2728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:45.334787 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:45.334805 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:45.335277 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:45.335634 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:45.335639 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:45.335643 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:45.335806 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:45.336269 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:45.336276 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:45.336603 (+ 327us) client_negotiation.cc:769] Sending connection context
0929 20:46:45.336825 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 20:46:45.337323 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":315,"threads_started":1}
I20260929 20:46:45.357642 2232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.344684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39155 (local address 127.0.97.62:36609)
0929 20:46:45.344862 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:45.344868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:45.345366 (+ 498us) server_negotiation.cc:407] Connection header received
0929 20:46:45.345560 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:45.345564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:45.345618 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:45.345691 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:45.346774 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.347533 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.349407 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.349744 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.349826 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.352026 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:45.352032 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:45.352037 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:45.352072 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:45.355079 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:45.355801 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:45.355807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:45.355811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:45.355957 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:45.356378 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:45.356382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:45.356384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:45.356678 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:45.356820 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:45.357249 (+ 429us) server_negotiation.cc:299] Negotiation successful
0929 20:46:45.357400 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:46:45.357959 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.344657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36609 (local address 127.0.97.3:39155)
0929 20:46:45.345259 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:45.345279 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:45.345444 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:45.345733 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:45.345738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:45.346308 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:45.346657 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.346668 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.347649 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.347655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.349273 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.349287 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.349903 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.349909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.350083 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.350679 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:45.350705 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:45.351837 (+ 1132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:45.355178 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:45.355183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:45.355187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:45.355675 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:45.356066 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:45.356071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:45.356074 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:45.356236 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:45.356806 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:45.356810 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:45.357000 (+ 190us) client_negotiation.cc:769] Sending connection context
0929 20:46:45.357163 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 20:46:45.357504 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":297,"threads_started":1}
I20260929 20:46:45.368182 2487 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46625
I20260929 20:46:45.368223 2621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46625 every 8 connection(s)
I20260929 20:46:45.369431 2487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:46:45.370961 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 2487
I20260929 20:46:45.371132 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:46:45.371279 2622 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36609
I20260929 20:46:45.371470 2622 heartbeater.cc:464] Registering TS with master...
I20260929 20:46:45.371613 388 external_mini_cluster.cc:1428] Setting key 7cbeccc99ae2a782cfd9c93b0eb1a232
I20260929 20:46:45.372123 2622 heartbeater.cc:511] Master 127.0.97.62:36609 requested a full tablet report, sending...
I20260929 20:46:45.373519 2136 ts_manager.cc:194] Registered new tserver with Master: ac1a30935b2647609e7a3b3eb055b6ad (127.0.97.3:46625)
I20260929 20:46:45.374462 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:46:45.374547 2136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44591
Sep 29 20:46:45 dist-test-slave-blv7 krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:46:45 dist-test-slave-blv7 krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:45.415316 2232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.402990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42234 (local address 127.0.97.62:36609)
0929 20:46:45.403221 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:45.403228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:45.403453 (+ 225us) server_negotiation.cc:407] Connection header received
0929 20:46:45.403627 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:45.403633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:45.403697 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:45.403776 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:45.405031 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.405739 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.406932 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.407380 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.407483 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.410796 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:45.410800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:45.410803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:45.410834 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:45.413035 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:45.413570 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:45.413575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:45.413579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:45.413700 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:45.414016 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:45.414021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:45.414025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:45.414264 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:45.414473 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:45.414761 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 20:46:45.414978 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 20:46:45.418902 2136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:46:45.425410 2136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:46:45.437315 2344 heartbeater.cc:503] Master 127.0.97.62:36609 was elected leader, sending a full tablet report...
I20260929 20:46:45.456673 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.437167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58828 (local address 127.0.97.2:45057)
0929 20:46:45.448493 (+ 11326us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:45.448501 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:45.448532 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:46:45.448626 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:45.448633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:45.448898 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:45.449111 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:46:45.449983 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.451304 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.453971 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.455509 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.455637 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.455841 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:45.456247 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 20:46:45.456402 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11163,"thread_start_us":11616,"threads_started":1}
I20260929 20:46:45.456702 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.436980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45057 (local address 127.0.0.1:58828)
0929 20:46:45.444194 (+ 7214us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:45.444249 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:45.444441 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:45.449352 (+ 4911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:45.449360 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:45.449388 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:46:45.449797 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.449810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.451453 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.451457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.453804 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.453818 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.455695 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.455701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.455878 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.455932 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 20:46:45.456071 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 20:46:45.456248 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6891,"mutex_wait_us":24,"thread_start_us":12109,"threads_started":1}
I20260929 20:46:45.456854 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.437581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44991 (local address 127.0.0.1:58668)
0929 20:46:45.449321 (+ 11740us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:45.449338 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:45.449442 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:45.450087 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:45.450091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:45.450113 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:46:45.450429 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.450440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.452240 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.452246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.454270 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.454280 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.456044 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.456051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.456201 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.456232 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:46:45.456315 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 20:46:45.456447 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11630,"mutex_wait_us":1,"thread_start_us":9739,"threads_started":1}
I20260929 20:46:45.457403 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.437757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37360 (local address 127.0.97.3:46625)
0929 20:46:45.449869 (+ 12112us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:45.449875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:45.449902 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:46:45.450003 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:45.450010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:45.450227 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:45.450422 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:46:45.451087 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.452494 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.454574 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.455941 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.456055 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.456261 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:45.456951 (+ 690us) server_negotiation.cc:299] Negotiation successful
0929 20:46:45.457101 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11890,"thread_start_us":11680,"threads_started":1}
I20260929 20:46:45.457436 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.438275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58668 (local address 127.0.97.1:44991)
0929 20:46:45.448101 (+ 9826us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:45.448109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:45.449564 (+ 1455us) server_negotiation.cc:407] Connection header received
0929 20:46:45.449656 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:45.449666 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:45.449922 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:45.450087 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:46:45.450603 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.452102 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.454449 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.455848 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.455969 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.456141 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:45.457049 (+ 908us) server_negotiation.cc:299] Negotiation successful
0929 20:46:45.457191 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9676,"thread_start_us":9714,"threads_started":1}
I20260929 20:46:45.457197 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.437239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46625 (local address 127.0.0.1:37360)
0929 20:46:45.447623 (+ 10384us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:45.447643 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:45.447754 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:45.450431 (+ 2677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:45.450435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:45.450457 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:46:45.450849 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.450861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.452617 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.452622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.454397 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.454411 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.456407 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.456414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.456587 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.456619 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:46:45.456710 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 20:46:45.456827 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10247,"mutex_wait_us":25,"thread_start_us":6480,"threads_started":1}
I20260929 20:46:45.460522 2413 tablet_service.cc:1467] Processing CreateTablet for tablet a49accef89094dd79d000cfddd400e13 (DEFAULT_TABLE table=test-table [id=2ba55858041f45f6874f85f53fe3b2f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:46:45.460577 2274 tablet_service.cc:1467] Processing CreateTablet for tablet a49accef89094dd79d000cfddd400e13 (DEFAULT_TABLE table=test-table [id=2ba55858041f45f6874f85f53fe3b2f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:46:45.461776 2413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a49accef89094dd79d000cfddd400e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:45.461777 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a49accef89094dd79d000cfddd400e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:45.463882 2552 tablet_service.cc:1467] Processing CreateTablet for tablet a49accef89094dd79d000cfddd400e13 (DEFAULT_TABLE table=test-table [id=2ba55858041f45f6874f85f53fe3b2f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:46:45.465117 2552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a49accef89094dd79d000cfddd400e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:45.472636 2641 tablet_bootstrap.cc:492] T a49accef89094dd79d000cfddd400e13 P 07c034afa3e44dcdaf52f96e4c4f14dc: Bootstrap starting.
I20260929 20:46:45.474929 2642 tablet_bootstrap.cc:492] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515: Bootstrap starting.
I20260929 20:46:45.475880 2641 tablet_bootstrap.cc:654] T a49accef89094dd79d000cfddd400e13 P 07c034afa3e44dcdaf52f96e4c4f14dc: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:45.477615 2641 log.cc:824] T a49accef89094dd79d000cfddd400e13 P 07c034afa3e44dcdaf52f96e4c4f14dc: Log is configured to *not* fsync() on all Append() calls
I20260929 20:46:45.478188 2642 tablet_bootstrap.cc:654] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:45.478493 2643 tablet_bootstrap.cc:492] T a49accef89094dd79d000cfddd400e13 P ac1a30935b2647609e7a3b3eb055b6ad: Bootstrap starting.
I20260929 20:46:45.479614 2642 log.cc:824] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515: Log is configured to *not* fsync() on all Append() calls
I20260929 20:46:45.481267 2641 tablet_bootstrap.cc:492] T a49accef89094dd79d000cfddd400e13 P 07c034afa3e44dcdaf52f96e4c4f14dc: No bootstrap required, opened a new log
I20260929 20:46:45.481405 2643 tablet_bootstrap.cc:654] T a49accef89094dd79d000cfddd400e13 P ac1a30935b2647609e7a3b3eb055b6ad: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:45.481529 2641 ts_tablet_manager.cc:1397] T a49accef89094dd79d000cfddd400e13 P 07c034afa3e44dcdaf52f96e4c4f14dc: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 20:46:45.482261 2643 log.cc:824] T a49accef89094dd79d000cfddd400e13 P ac1a30935b2647609e7a3b3eb055b6ad: Log is configured to *not* fsync() on all Append() calls
I20260929 20:46:45.482287 2642 tablet_bootstrap.cc:492] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515: No bootstrap required, opened a new log
I20260929 20:46:45.482509 2642 ts_tablet_manager.cc:1397] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 20:46:45.484521 2643 tablet_bootstrap.cc:492] T a49accef89094dd79d000cfddd400e13 P ac1a30935b2647609e7a3b3eb055b6ad: No bootstrap required, opened a new log
I20260929 20:46:45.484735 2643 ts_tablet_manager.cc:1397] T a49accef89094dd79d000cfddd400e13 P ac1a30935b2647609e7a3b3eb055b6ad: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 20:46:45.491416 2641 raft_consensus.cc:348] T a49accef89094dd79d000cfddd400e13 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } } peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } }
I20260929 20:46:45.491814 2641 raft_consensus.cc:374] T a49accef89094dd79d000cfddd400e13 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:45.491972 2641 raft_consensus.cc:729] T a49accef89094dd79d000cfddd400e13 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c034afa3e44dcdaf52f96e4c4f14dc, State: Initialized, Role: FOLLOWER
I20260929 20:46:45.492079 2643 raft_consensus.cc:348] T a49accef89094dd79d000cfddd400e13 P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } } peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } }
I20260929 20:46:45.492411 2643 raft_consensus.cc:374] T a49accef89094dd79d000cfddd400e13 P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:45.492188 2642 raft_consensus.cc:348] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } } peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } }
I20260929 20:46:45.492507 2643 raft_consensus.cc:729] T a49accef89094dd79d000cfddd400e13 P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1a30935b2647609e7a3b3eb055b6ad, State: Initialized, Role: FOLLOWER
I20260929 20:46:45.492545 2642 raft_consensus.cc:374] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:45.492621 2642 raft_consensus.cc:729] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed8101318574e84b35484a916fa5515, State: Initialized, Role: FOLLOWER
I20260929 20:46:45.492642 2641 consensus_queue.cc:260] T a49accef89094dd79d000cfddd400e13 P 07c034afa3e44dcdaf52f96e4c4f14dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } } peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } }
I20260929 20:46:45.493126 2643 consensus_queue.cc:260] T a49accef89094dd79d000cfddd400e13 P ac1a30935b2647609e7a3b3eb055b6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } } peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } }
I20260929 20:46:45.493175 2642 consensus_queue.cc:260] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } } peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } }
I20260929 20:46:45.494014 2622 heartbeater.cc:503] Master 127.0.97.62:36609 was elected leader, sending a full tablet report...
I20260929 20:46:45.497857 2643 ts_tablet_manager.cc:1428] T a49accef89094dd79d000cfddd400e13 P ac1a30935b2647609e7a3b3eb055b6ad: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260929 20:46:45.500581 2641 ts_tablet_manager.cc:1428] T a49accef89094dd79d000cfddd400e13 P 07c034afa3e44dcdaf52f96e4c4f14dc: Time spent starting tablet: real 0.019s user 0.009s sys 0.010s
I20260929 20:46:45.500919 2483 heartbeater.cc:503] Master 127.0.97.62:36609 was elected leader, sending a full tablet report...
I20260929 20:46:45.501857 2642 ts_tablet_manager.cc:1428] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515: Time spent starting tablet: real 0.019s user 0.011s sys 0.008s
I20260929 20:46:45.602619 2649 raft_consensus.cc:482] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:46:45.602880 2649 raft_consensus.cc:504] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } } peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } }
I20260929 20:46:45.604597 2649 leader_election.cc:288] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07c034afa3e44dcdaf52f96e4c4f14dc (127.0.97.2:45057), ac1a30935b2647609e7a3b3eb055b6ad (127.0.97.3:46625)
I20260929 20:46:45.612901 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.605028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38705 (local address 127.0.97.3:46625)
0929 20:46:45.605404 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:45.605410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:45.605862 (+ 452us) server_negotiation.cc:407] Connection header received
0929 20:46:45.605988 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:45.605993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:45.606057 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:45.606133 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:46:45.606871 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.607931 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.610742 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.611944 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.612070 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.612171 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:45.612545 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 20:46:45.612663 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244}
I20260929 20:46:45.612908 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.605021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38631 (local address 127.0.97.2:45057)
0929 20:46:45.605237 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:45.605244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:45.605692 (+ 448us) server_negotiation.cc:407] Connection header received
0929 20:46:45.605930 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:45.605935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:45.606010 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:45.606107 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:46:45.606830 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.607935 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.610742 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.611942 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.612070 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.612171 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:45.612545 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 20:46:45.612663 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 20:46:45.613114 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.605009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46625 (local address 127.0.97.1:38705)
0929 20:46:45.605778 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:45.605792 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:45.605889 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:45.606145 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:45.606149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:45.606172 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:46:45.606725 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.606735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.608063 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.608068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.610601 (+ 2533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.610610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.612148 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.612154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.612388 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.612428 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 20:46:45.612517 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 20:46:45.612704 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":239,"thread_start_us":520,"threads_started":1}
I20260929 20:46:45.613147 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.604782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45057 (local address 127.0.97.1:38631)
0929 20:46:45.605572 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:45.605594 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:45.605749 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:45.606141 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:45.606145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:45.606225 (+ 80us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:46:45.606659 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.606676 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.608064 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.608068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.610582 (+ 2514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.610602 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.612176 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.612182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.612382 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.612420 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 20:46:45.612517 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 20:46:45.612703 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":235,"spinlock_wait_cycles":23808,"thread_start_us":467,"threads_started":1}
I20260929 20:46:45.614094 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a49accef89094dd79d000cfddd400e13" candidate_uuid: "3ed8101318574e84b35484a916fa5515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" is_pre_election: true
I20260929 20:46:45.614089 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a49accef89094dd79d000cfddd400e13" candidate_uuid: "3ed8101318574e84b35484a916fa5515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" is_pre_election: true
I20260929 20:46:45.614509 2577 raft_consensus.cc:2454] T a49accef89094dd79d000cfddd400e13 P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ed8101318574e84b35484a916fa5515 in term 0.
I20260929 20:46:45.614511 2438 raft_consensus.cc:2454] T a49accef89094dd79d000cfddd400e13 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ed8101318574e84b35484a916fa5515 in term 0.
I20260929 20:46:45.615094 2224 leader_election.cc:302] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed8101318574e84b35484a916fa5515, ac1a30935b2647609e7a3b3eb055b6ad; no voters:
I20260929 20:46:45.615451 2649 raft_consensus.cc:2781] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:46:45.615566 2649 raft_consensus.cc:482] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:46:45.615646 2649 raft_consensus.cc:3037] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:45.617779 2649 raft_consensus.cc:504] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } } peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } }
I20260929 20:46:45.618285 2649 leader_election.cc:288] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515 [CANDIDATE]: Term 1 election: Requested vote from peers 07c034afa3e44dcdaf52f96e4c4f14dc (127.0.97.2:45057), ac1a30935b2647609e7a3b3eb055b6ad (127.0.97.3:46625)
I20260929 20:46:45.618741 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a49accef89094dd79d000cfddd400e13" candidate_uuid: "3ed8101318574e84b35484a916fa5515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac1a30935b2647609e7a3b3eb055b6ad"
I20260929 20:46:45.618741 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a49accef89094dd79d000cfddd400e13" candidate_uuid: "3ed8101318574e84b35484a916fa5515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc"
I20260929 20:46:45.618942 2438 raft_consensus.cc:3037] T a49accef89094dd79d000cfddd400e13 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:45.618930 2577 raft_consensus.cc:3037] T a49accef89094dd79d000cfddd400e13 P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Advancing to term 1
W20260929 20:46:45.620810 2623 tablet.cc:2369] T a49accef89094dd79d000cfddd400e13 P ac1a30935b2647609e7a3b3eb055b6ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:46:45.622007 2577 raft_consensus.cc:2454] T a49accef89094dd79d000cfddd400e13 P ac1a30935b2647609e7a3b3eb055b6ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed8101318574e84b35484a916fa5515 in term 1.
I20260929 20:46:45.622007 2438 raft_consensus.cc:2454] T a49accef89094dd79d000cfddd400e13 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed8101318574e84b35484a916fa5515 in term 1.
I20260929 20:46:45.622406 2226 leader_election.cc:302] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07c034afa3e44dcdaf52f96e4c4f14dc, 3ed8101318574e84b35484a916fa5515, ac1a30935b2647609e7a3b3eb055b6ad; no voters:
I20260929 20:46:45.622701 2649 raft_consensus.cc:2781] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:46:45.623116 2649 raft_consensus.cc:686] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515 [term 1 LEADER]: Becoming Leader. State: Replica: 3ed8101318574e84b35484a916fa5515, State: Running, Role: LEADER
I20260929 20:46:45.623535 2649 consensus_queue.cc:237] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } } peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } }
I20260929 20:46:45.627555 2137 catalog_manager.cc:5495] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ed8101318574e84b35484a916fa5515 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "3ed8101318574e84b35484a916fa5515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } health_report { overall_health: HEALTHY } } }
W20260929 20:46:45.651635 2484 tablet.cc:2369] T a49accef89094dd79d000cfddd400e13 P 07c034afa3e44dcdaf52f96e4c4f14dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 20:46:45 dist-test-slave-blv7 krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:45.673225 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.661198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36609 (local address 127.0.0.1:42242)
0929 20:46:45.661756 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:45.661771 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:45.661945 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:45.662230 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:45.662234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:45.662871 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:45.663151 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.663161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.664116 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.664121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.665080 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.665091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.665767 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.665777 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.665933 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.666456 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:45.666472 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:45.668511 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:45.671109 (+ 2598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:45.671118 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:45.671123 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:45.671506 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:45.671820 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:45.671823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:45.671825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:45.671957 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:45.672451 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:45.672458 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:45.672588 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 20:46:45.672721 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 20:46:45.672913 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":306,"threads_started":1}
I20260929 20:46:45.673261 2232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.661163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42242 (local address 127.0.97.62:36609)
0929 20:46:45.661319 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:45.661327 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:45.661852 (+ 525us) server_negotiation.cc:407] Connection header received
0929 20:46:45.662060 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:45.662065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:45.662120 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:45.662194 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:45.663289 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.664003 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.665203 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.665611 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.665712 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.668660 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:45.668666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:45.668671 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:45.668724 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:45.670968 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:45.671601 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:45.671604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:45.671606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:45.671723 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:45.672041 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:45.672065 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:45.672069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:45.672299 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:45.672397 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:45.672803 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 20:46:45.673016 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":49}
I20260929 20:46:45.677608 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42242:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 20:46:45.678563 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260929 20:46:45.680253 2345 tablet.cc:2369] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:46:45.684859 2413 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc050d592d74508a0aec08103e45931 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=120f6c01803c40cfb8843dc5f48e57d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:46:45.684741 2274 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc050d592d74508a0aec08103e45931 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=120f6c01803c40cfb8843dc5f48e57d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:46:45.685088 2552 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc050d592d74508a0aec08103e45931 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=120f6c01803c40cfb8843dc5f48e57d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:46:45.685635 2413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc050d592d74508a0aec08103e45931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:45.685698 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc050d592d74508a0aec08103e45931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:45.685819 2552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc050d592d74508a0aec08103e45931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:45.690187 2641 tablet_bootstrap.cc:492] T 8fc050d592d74508a0aec08103e45931 P 07c034afa3e44dcdaf52f96e4c4f14dc: Bootstrap starting.
I20260929 20:46:45.691746 2641 tablet_bootstrap.cc:654] T 8fc050d592d74508a0aec08103e45931 P 07c034afa3e44dcdaf52f96e4c4f14dc: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:45.692413 2642 tablet_bootstrap.cc:492] T 8fc050d592d74508a0aec08103e45931 P 3ed8101318574e84b35484a916fa5515: Bootstrap starting.
I20260929 20:46:45.692577 2643 tablet_bootstrap.cc:492] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad: Bootstrap starting.
I20260929 20:46:45.693783 2643 tablet_bootstrap.cc:654] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:45.694065 2642 tablet_bootstrap.cc:654] T 8fc050d592d74508a0aec08103e45931 P 3ed8101318574e84b35484a916fa5515: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:45.696139 2642 tablet_bootstrap.cc:492] T 8fc050d592d74508a0aec08103e45931 P 3ed8101318574e84b35484a916fa5515: No bootstrap required, opened a new log
I20260929 20:46:45.696264 2642 ts_tablet_manager.cc:1397] T 8fc050d592d74508a0aec08103e45931 P 3ed8101318574e84b35484a916fa5515: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260929 20:46:45.696337 2643 tablet_bootstrap.cc:492] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad: No bootstrap required, opened a new log
I20260929 20:46:45.696444 2643 ts_tablet_manager.cc:1397] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260929 20:46:45.696973 2642 raft_consensus.cc:348] T 8fc050d592d74508a0aec08103e45931 P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:45.697185 2642 raft_consensus.cc:374] T 8fc050d592d74508a0aec08103e45931 P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:45.697067 2643 raft_consensus.cc:348] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:45.697263 2643 raft_consensus.cc:374] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:45.697265 2642 raft_consensus.cc:729] T 8fc050d592d74508a0aec08103e45931 P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed8101318574e84b35484a916fa5515, State: Initialized, Role: FOLLOWER
I20260929 20:46:45.697350 2643 raft_consensus.cc:729] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1a30935b2647609e7a3b3eb055b6ad, State: Initialized, Role: FOLLOWER
I20260929 20:46:45.697484 2642 consensus_queue.cc:260] T 8fc050d592d74508a0aec08103e45931 P 3ed8101318574e84b35484a916fa5515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:45.697541 2643 consensus_queue.cc:260] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:45.697894 2649 tablet_replica.cc:439] T 8fc050d592d74508a0aec08103e45931 P 3ed8101318574e84b35484a916fa5515: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } } }
I20260929 20:46:45.698076 2642 ts_tablet_manager.cc:1428] T 8fc050d592d74508a0aec08103e45931 P 3ed8101318574e84b35484a916fa5515: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 20:46:45.697918 2648 tablet_replica.cc:439] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } } }
I20260929 20:46:45.698174 2649 tablet_replica.cc:442] T 8fc050d592d74508a0aec08103e45931 P 3ed8101318574e84b35484a916fa5515: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:46:45.698210 2648 tablet_replica.cc:442] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:46:45.698974 2643 ts_tablet_manager.cc:1428] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 20:46:45.699548 2641 tablet_bootstrap.cc:492] T 8fc050d592d74508a0aec08103e45931 P 07c034afa3e44dcdaf52f96e4c4f14dc: No bootstrap required, opened a new log
I20260929 20:46:45.699738 2641 ts_tablet_manager.cc:1397] T 8fc050d592d74508a0aec08103e45931 P 07c034afa3e44dcdaf52f96e4c4f14dc: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260929 20:46:45.700492 2641 raft_consensus.cc:348] T 8fc050d592d74508a0aec08103e45931 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:45.700781 2641 raft_consensus.cc:374] T 8fc050d592d74508a0aec08103e45931 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:45.700892 2641 raft_consensus.cc:729] T 8fc050d592d74508a0aec08103e45931 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c034afa3e44dcdaf52f96e4c4f14dc, State: Initialized, Role: FOLLOWER
I20260929 20:46:45.701169 2641 consensus_queue.cc:260] T 8fc050d592d74508a0aec08103e45931 P 07c034afa3e44dcdaf52f96e4c4f14dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:45.701767 2641 ts_tablet_manager.cc:1428] T 8fc050d592d74508a0aec08103e45931 P 07c034afa3e44dcdaf52f96e4c4f14dc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:46:45.701658 2647 tablet_replica.cc:439] T 8fc050d592d74508a0aec08103e45931 P 07c034afa3e44dcdaf52f96e4c4f14dc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } } }
I20260929 20:46:45.702010 2647 tablet_replica.cc:442] T 8fc050d592d74508a0aec08103e45931 P 07c034afa3e44dcdaf52f96e4c4f14dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:46:45.763479 2648 raft_consensus.cc:482] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:46:45.763706 2648 raft_consensus.cc:504] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:45.765205 2648 leader_election.cc:288] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ed8101318574e84b35484a916fa5515 (127.0.97.1:44991), 07c034afa3e44dcdaf52f96e4c4f14dc (127.0.97.2:45057)
I20260929 20:46:45.770733 2511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.765541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45057 (local address 127.0.97.3:53943)
0929 20:46:45.765710 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:45.765731 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:45.765840 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:45.766192 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:45.766196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:45.766221 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:46:45.766458 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.766477 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.767654 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.767658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.769013 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.769027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.770070 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.770075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.770235 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.770258 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:46:45.770336 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 20:46:45.770460 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 20:46:45.770750 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.765655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53943 (local address 127.0.97.2:45057)
0929 20:46:45.765837 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:45.765842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:45.765860 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:46:45.765952 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:45.765957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:45.766043 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:45.766138 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:46:45.766609 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.767530 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.769155 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.769925 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.770035 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.770121 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:45.770366 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 20:46:45.770492 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:46:45.771494 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc050d592d74508a0aec08103e45931" candidate_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" is_pre_election: true
I20260929 20:46:45.771739 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.765558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44991 (local address 127.0.97.3:55475)
0929 20:46:45.766705 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:45.766722 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:45.766822 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:45.767200 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:45.767204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:45.767226 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:46:45.767500 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.767508 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.768533 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.768538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.770000 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.770012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.771108 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.771114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.771265 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.771295 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:46:45.771369 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 20:46:45.771460 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"mutex_wait_us":374,"thread_start_us":439,"threads_started":1}
I20260929 20:46:45.771740 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.765642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55475 (local address 127.0.97.1:44991)
0929 20:46:45.765813 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:45.765817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:45.766797 (+ 980us) server_negotiation.cc:407] Connection header received
0929 20:46:45.766988 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:45.766991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:45.767092 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:45.767163 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:46:45.767629 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.768410 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.770134 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.770878 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.771039 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.771126 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:45.771415 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 20:46:45.771533 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:46:45.771757 2438 raft_consensus.cc:2454] T 8fc050d592d74508a0aec08103e45931 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac1a30935b2647609e7a3b3eb055b6ad in term 0.
I20260929 20:46:45.772207 2505 leader_election.cc:302] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c034afa3e44dcdaf52f96e4c4f14dc, ac1a30935b2647609e7a3b3eb055b6ad; no voters:
I20260929 20:46:45.772641 2648 raft_consensus.cc:2781] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:46:45.772749 2648 raft_consensus.cc:482] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:46:45.772811 2648 raft_consensus.cc:3037] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:45.773097 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc050d592d74508a0aec08103e45931" candidate_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed8101318574e84b35484a916fa5515" is_pre_election: true
I20260929 20:46:45.773516 2294 raft_consensus.cc:2454] T 8fc050d592d74508a0aec08103e45931 P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac1a30935b2647609e7a3b3eb055b6ad in term 0.
I20260929 20:46:45.774994 2648 raft_consensus.cc:504] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:45.775534 2648 leader_election.cc:288] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad [CANDIDATE]: Term 1 election: Requested vote from peers 3ed8101318574e84b35484a916fa5515 (127.0.97.1:44991), 07c034afa3e44dcdaf52f96e4c4f14dc (127.0.97.2:45057)
I20260929 20:46:45.775965 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc050d592d74508a0aec08103e45931" candidate_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed8101318574e84b35484a916fa5515"
I20260929 20:46:45.775966 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc050d592d74508a0aec08103e45931" candidate_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc"
I20260929 20:46:45.776144 2294 raft_consensus.cc:3037] T 8fc050d592d74508a0aec08103e45931 P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:45.776142 2438 raft_consensus.cc:3037] T 8fc050d592d74508a0aec08103e45931 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:45.778096 2438 raft_consensus.cc:2454] T 8fc050d592d74508a0aec08103e45931 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac1a30935b2647609e7a3b3eb055b6ad in term 1.
I20260929 20:46:45.778290 2294 raft_consensus.cc:2454] T 8fc050d592d74508a0aec08103e45931 P 3ed8101318574e84b35484a916fa5515 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac1a30935b2647609e7a3b3eb055b6ad in term 1.
I20260929 20:46:45.778426 2505 leader_election.cc:302] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c034afa3e44dcdaf52f96e4c4f14dc, ac1a30935b2647609e7a3b3eb055b6ad; no voters:
I20260929 20:46:45.778704 2648 raft_consensus.cc:2781] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:46:45.779186 2648 raft_consensus.cc:686] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad [term 1 LEADER]: Becoming Leader. State: Replica: ac1a30935b2647609e7a3b3eb055b6ad, State: Running, Role: LEADER
I20260929 20:46:45.779726 2648 consensus_queue.cc:237] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:45.782835 2664 tablet_replica.cc:439] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad: TxnStatusTablet state changed. Reason: New leader ac1a30935b2647609e7a3b3eb055b6ad. Latest consensus state: current_term: 1 leader_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } } }
I20260929 20:46:45.783241 2664 tablet_replica.cc:442] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:46:45.783898 2666 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 20:46:45.784157 2666 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 20:46:45.784099 2137 catalog_manager.cc:5495] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad reported cstate change: term changed from 0 to 1, leader changed from <none> to ac1a30935b2647609e7a3b3eb055b6ad (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } health_report { overall_health: HEALTHY } } }
I20260929 20:46:45.805562 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.799745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46625 (local address 127.0.0.1:37366)
0929 20:46:45.799952 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:45.799972 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:45.800100 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:45.800544 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:45.800557 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:45.800590 (+ 33us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:46:45.800766 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.800774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.802189 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.802193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.803192 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.803204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.804107 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.804111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.804235 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.804282 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:46:45.805180 (+ 898us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:46:45.805189 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 20:46:45.805231 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 20:46:45.805329 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 20:46:45.805644 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.799748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.0.97.3:46625)
0929 20:46:45.799938 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:45.799943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:45.800033 (+ 90us) server_negotiation.cc:407] Connection header received
0929 20:46:45.800274 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:45.800279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:45.800374 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:45.800529 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:46:45.800896 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.802072 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.803315 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.803961 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.804102 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.804445 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:46:45.805092 (+ 647us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:46:45.805226 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:45.805269 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 20:46:45.805352 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:46:45.813222 2294 raft_consensus.cc:1261] T 8fc050d592d74508a0aec08103e45931 P 3ed8101318574e84b35484a916fa5515 [term 1 FOLLOWER]: Refusing update from remote peer ac1a30935b2647609e7a3b3eb055b6ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:46:45.813256 2438 raft_consensus.cc:1261] T 8fc050d592d74508a0aec08103e45931 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 1 FOLLOWER]: Refusing update from remote peer ac1a30935b2647609e7a3b3eb055b6ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:46:45.814097 2664 consensus_queue.cc:1035] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:46:45.814363 2648 consensus_queue.cc:1035] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:46:45.821352 2647 tablet_replica.cc:439] T 8fc050d592d74508a0aec08103e45931 P 07c034afa3e44dcdaf52f96e4c4f14dc: TxnStatusTablet state changed. Reason: New leader ac1a30935b2647609e7a3b3eb055b6ad. Latest consensus state: current_term: 1 leader_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } } }
I20260929 20:46:45.821640 2647 tablet_replica.cc:442] T 8fc050d592d74508a0aec08103e45931 P 07c034afa3e44dcdaf52f96e4c4f14dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:46:45.833081 2647 tablet_replica.cc:439] T 8fc050d592d74508a0aec08103e45931 P 07c034afa3e44dcdaf52f96e4c4f14dc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } } }
I20260929 20:46:45.833357 2647 tablet_replica.cc:442] T 8fc050d592d74508a0aec08103e45931 P 07c034afa3e44dcdaf52f96e4c4f14dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:46:45.837167 2649 tablet_replica.cc:439] T 8fc050d592d74508a0aec08103e45931 P 3ed8101318574e84b35484a916fa5515: TxnStatusTablet state changed. Reason: New leader ac1a30935b2647609e7a3b3eb055b6ad. Latest consensus state: current_term: 1 leader_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } } }
I20260929 20:46:45.837494 2649 tablet_replica.cc:442] T 8fc050d592d74508a0aec08103e45931 P 3ed8101318574e84b35484a916fa5515: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:46:45.843480 2649 tablet_replica.cc:439] T 8fc050d592d74508a0aec08103e45931 P 3ed8101318574e84b35484a916fa5515: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } } }
I20260929 20:46:45.843732 2649 tablet_replica.cc:442] T 8fc050d592d74508a0aec08103e45931 P 3ed8101318574e84b35484a916fa5515: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:46:45.851998 2648 tablet_replica.cc:439] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } } }
I20260929 20:46:45.852293 2648 tablet_replica.cc:442] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:46:45.852684 2664 tablet_replica.cc:439] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } } }
I20260929 20:46:45.852993 2664 tablet_replica.cc:442] T 8fc050d592d74508a0aec08103e45931 P ac1a30935b2647609e7a3b3eb055b6ad: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:46:45.865051 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.859374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58680 (local address 127.0.97.1:44991)
0929 20:46:45.859545 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:45.859551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:45.859706 (+ 155us) server_negotiation.cc:407] Connection header received
0929 20:46:45.859912 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:45.859915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:45.859983 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:45.860096 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:46:45.860444 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.861161 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.863079 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.863455 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.863544 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.864024 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:46:45.864515 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:46:45.864609 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:45.864722 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 20:46:45.864780 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:46:45.880213 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.874557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46625 (local address 127.0.97.1:43971)
0929 20:46:45.875210 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:45.875224 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:45.875323 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:45.875629 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:45.875632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:45.875640 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:46:45.875918 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.875925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.877206 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.877210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.878277 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.878288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.879196 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.879201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.879304 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.879352 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:46:45.879788 (+ 436us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:46:45.879797 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 20:46:45.879854 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 20:46:45.879957 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":345,"threads_started":1}
I20260929 20:46:45.880247 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.874577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43971 (local address 127.0.97.3:46625)
0929 20:46:45.874743 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:45.874749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:45.875291 (+ 542us) server_negotiation.cc:407] Connection header received
0929 20:46:45.875411 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:45.875417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:45.875536 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:45.875626 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:46:45.876041 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.877103 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.878394 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.879049 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.879159 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.879561 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:46:45.879696 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:46:45.879779 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:45.879897 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 20:46:45.879982 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 20:46:45.888773 2438 raft_consensus.cc:1261] T a49accef89094dd79d000cfddd400e13 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 1 FOLLOWER]: Refusing update from remote peer 3ed8101318574e84b35484a916fa5515: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:46:45.888995 2577 raft_consensus.cc:1261] T a49accef89094dd79d000cfddd400e13 P ac1a30935b2647609e7a3b3eb055b6ad [term 1 FOLLOWER]: Refusing update from remote peer 3ed8101318574e84b35484a916fa5515: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:46:45.889299 2652 consensus_queue.cc:1035] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:46:45.889684 2649 consensus_queue.cc:1035] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:46:45.904297 2669 mvcc.cc:205] Tried to move back new op lower bound from 7334767844915707904 to 7334767843837235200. Current Snapshot: MvccSnapshot[applied={T|T < 7334767844915707904}]
I20260929 20:46:45.933307 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.925411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57559 (local address 127.0.97.1:44991)
0929 20:46:45.925562 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:45.925568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:45.926747 (+ 1179us) server_negotiation.cc:407] Connection header received
0929 20:46:45.926919 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:45.926935 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:45.927059 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:45.927142 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:46:45.927882 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.929027 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.931137 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.931738 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.931831 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.932736 (+ 905us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:46:45.932839 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:46:45.932911 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:45.933061 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 20:46:45.933119 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 20:46:45.933606 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:45.925348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44991 (local address 127.0.97.3:57559)
0929 20:46:45.926644 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:45.926660 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:45.926792 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:45.927335 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:45.927340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:45.927354 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:46:45.927713 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.927724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.929146 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.929151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.930959 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:45.930973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:45.932262 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:45.932268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:45.932437 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:45.932507 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:46:45.932944 (+ 437us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:46:45.932956 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 20:46:45.933050 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 20:46:45.933197 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"thread_start_us":939,"threads_started":1}
I20260929 20:46:45.967195 2135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:46:45.967490 2135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:46:45.971271 2135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052: Sending DeleteTablet for 3 replicas of tablet a49accef89094dd79d000cfddd400e13
I20260929 20:46:45.972375 2413 tablet_service.cc:1514] Processing DeleteTablet for tablet a49accef89094dd79d000cfddd400e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:46:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58828
I20260929 20:46:45.972743 2274 tablet_service.cc:1514] Processing DeleteTablet for tablet a49accef89094dd79d000cfddd400e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:46:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58668
I20260929 20:46:45.974479 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:46:45.975095 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:46:45.977802 2552 tablet_service.cc:1514] Processing DeleteTablet for tablet a49accef89094dd79d000cfddd400e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:46:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37360
I20260929 20:46:45.981930 2412 tablet_service.cc:1467] Processing CreateTablet for tablet 07a89b147e6c43c898db2f126cdd6a5a (DEFAULT_TABLE table=test-table [id=992c3f87bc60492e87221c76e2e03bfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:46:45.982393 2412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a89b147e6c43c898db2f126cdd6a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:45.987509 2273 tablet_service.cc:1467] Processing CreateTablet for tablet 07a89b147e6c43c898db2f126cdd6a5a (DEFAULT_TABLE table=test-table [id=992c3f87bc60492e87221c76e2e03bfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:46:45.987996 2273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a89b147e6c43c898db2f126cdd6a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:45.991549 2551 tablet_service.cc:1467] Processing CreateTablet for tablet 07a89b147e6c43c898db2f126cdd6a5a (DEFAULT_TABLE table=test-table [id=992c3f87bc60492e87221c76e2e03bfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:46:45.992048 2551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a89b147e6c43c898db2f126cdd6a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:46.004087 2642 tablet_bootstrap.cc:492] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515: Bootstrap starting.
I20260929 20:46:46.004300 2641 tablet_bootstrap.cc:492] T 07a89b147e6c43c898db2f126cdd6a5a P 07c034afa3e44dcdaf52f96e4c4f14dc: Bootstrap starting.
I20260929 20:46:46.005733 2642 tablet_bootstrap.cc:654] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:46.005729 2641 tablet_bootstrap.cc:654] T 07a89b147e6c43c898db2f126cdd6a5a P 07c034afa3e44dcdaf52f96e4c4f14dc: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:46.009263 2643 tablet_bootstrap.cc:492] T 07a89b147e6c43c898db2f126cdd6a5a P ac1a30935b2647609e7a3b3eb055b6ad: Bootstrap starting.
I20260929 20:46:46.011062 2643 tablet_bootstrap.cc:654] T 07a89b147e6c43c898db2f126cdd6a5a P ac1a30935b2647609e7a3b3eb055b6ad: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:46.011428 2691 tablet_replica.cc:330] T a49accef89094dd79d000cfddd400e13 P ac1a30935b2647609e7a3b3eb055b6ad: stopping tablet replica
I20260929 20:46:46.011876 2691 raft_consensus.cc:2229] T a49accef89094dd79d000cfddd400e13 P ac1a30935b2647609e7a3b3eb055b6ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:46:46.012207 2691 raft_consensus.cc:2258] T a49accef89094dd79d000cfddd400e13 P ac1a30935b2647609e7a3b3eb055b6ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:46:46.012828 2690 tablet_replica.cc:330] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515: stopping tablet replica
I20260929 20:46:46.013294 2690 raft_consensus.cc:2229] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:46:46.013985 2690 raft_consensus.cc:2258] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:46:46.019096 2690 ts_tablet_manager.cc:1905] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:46:46.016577 2691 ts_tablet_manager.cc:1905] T a49accef89094dd79d000cfddd400e13 P ac1a30935b2647609e7a3b3eb055b6ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:46:46.021989 2643 tablet_bootstrap.cc:492] T 07a89b147e6c43c898db2f126cdd6a5a P ac1a30935b2647609e7a3b3eb055b6ad: No bootstrap required, opened a new log
I20260929 20:46:46.022239 2643 ts_tablet_manager.cc:1397] T 07a89b147e6c43c898db2f126cdd6a5a P ac1a30935b2647609e7a3b3eb055b6ad: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.004s
I20260929 20:46:46.023128 2643 raft_consensus.cc:348] T 07a89b147e6c43c898db2f126cdd6a5a P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:46.023512 2643 raft_consensus.cc:374] T 07a89b147e6c43c898db2f126cdd6a5a P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:46.023633 2643 raft_consensus.cc:729] T 07a89b147e6c43c898db2f126cdd6a5a P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1a30935b2647609e7a3b3eb055b6ad, State: Initialized, Role: FOLLOWER
I20260929 20:46:46.023968 2643 consensus_queue.cc:260] T 07a89b147e6c43c898db2f126cdd6a5a P ac1a30935b2647609e7a3b3eb055b6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:46.024645 2643 ts_tablet_manager.cc:1428] T 07a89b147e6c43c898db2f126cdd6a5a P ac1a30935b2647609e7a3b3eb055b6ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:46:46.033385 2690 ts_tablet_manager.cc:1918] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:46:46.033484 2641 tablet_bootstrap.cc:492] T 07a89b147e6c43c898db2f126cdd6a5a P 07c034afa3e44dcdaf52f96e4c4f14dc: No bootstrap required, opened a new log
I20260929 20:46:46.033658 2641 ts_tablet_manager.cc:1397] T 07a89b147e6c43c898db2f126cdd6a5a P 07c034afa3e44dcdaf52f96e4c4f14dc: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.006s
I20260929 20:46:46.034431 2641 raft_consensus.cc:348] T 07a89b147e6c43c898db2f126cdd6a5a P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:46.034673 2641 raft_consensus.cc:374] T 07a89b147e6c43c898db2f126cdd6a5a P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:46.034751 2641 raft_consensus.cc:729] T 07a89b147e6c43c898db2f126cdd6a5a P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c034afa3e44dcdaf52f96e4c4f14dc, State: Initialized, Role: FOLLOWER
I20260929 20:46:46.035181 2641 consensus_queue.cc:260] T 07a89b147e6c43c898db2f126cdd6a5a P 07c034afa3e44dcdaf52f96e4c4f14dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:46.036022 2641 ts_tablet_manager.cc:1428] T 07a89b147e6c43c898db2f126cdd6a5a P 07c034afa3e44dcdaf52f96e4c4f14dc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 20:46:46.037025 2689 tablet_replica.cc:330] T a49accef89094dd79d000cfddd400e13 P 07c034afa3e44dcdaf52f96e4c4f14dc: stopping tablet replica
I20260929 20:46:46.037557 2689 raft_consensus.cc:2229] T a49accef89094dd79d000cfddd400e13 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:46:46.038442 2689 raft_consensus.cc:2258] T a49accef89094dd79d000cfddd400e13 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:46:46.040421 2689 ts_tablet_manager.cc:1905] T a49accef89094dd79d000cfddd400e13 P 07c034afa3e44dcdaf52f96e4c4f14dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:46:46.038812 2642 tablet_bootstrap.cc:492] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515: No bootstrap required, opened a new log
I20260929 20:46:46.040990 2690 log.cc:1196] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/a49accef89094dd79d000cfddd400e13
I20260929 20:46:46.041064 2642 ts_tablet_manager.cc:1397] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.012s
I20260929 20:46:46.041532 2690 ts_tablet_manager.cc:1939] T a49accef89094dd79d000cfddd400e13 P 3ed8101318574e84b35484a916fa5515: Deleting consensus metadata
I20260929 20:46:46.041891 2691 ts_tablet_manager.cc:1918] T a49accef89094dd79d000cfddd400e13 P ac1a30935b2647609e7a3b3eb055b6ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:46:46.041862 2642 raft_consensus.cc:348] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:46.042088 2642 raft_consensus.cc:374] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:46.042158 2642 raft_consensus.cc:729] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed8101318574e84b35484a916fa5515, State: Initialized, Role: FOLLOWER
I20260929 20:46:46.042266 2691 log.cc:1196] T a49accef89094dd79d000cfddd400e13 P ac1a30935b2647609e7a3b3eb055b6ad: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/a49accef89094dd79d000cfddd400e13
I20260929 20:46:46.042374 2642 consensus_queue.cc:260] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:46.042932 2642 ts_tablet_manager.cc:1428] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 20:46:46.046593 2691 ts_tablet_manager.cc:1939] T a49accef89094dd79d000cfddd400e13 P ac1a30935b2647609e7a3b3eb055b6ad: Deleting consensus metadata
I20260929 20:46:46.047730 2689 ts_tablet_manager.cc:1918] T a49accef89094dd79d000cfddd400e13 P 07c034afa3e44dcdaf52f96e4c4f14dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:46:46.047932 2689 log.cc:1196] T a49accef89094dd79d000cfddd400e13 P 07c034afa3e44dcdaf52f96e4c4f14dc: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/a49accef89094dd79d000cfddd400e13
I20260929 20:46:46.048151 2124 catalog_manager.cc:4841] TS 3ed8101318574e84b35484a916fa5515 (127.0.97.1:44991): tablet a49accef89094dd79d000cfddd400e13 (table test-table [id=2ba55858041f45f6874f85f53fe3b2f6]) successfully deleted
I20260929 20:46:46.048453 2689 ts_tablet_manager.cc:1939] T a49accef89094dd79d000cfddd400e13 P 07c034afa3e44dcdaf52f96e4c4f14dc: Deleting consensus metadata
I20260929 20:46:46.049803 2121 catalog_manager.cc:4841] TS ac1a30935b2647609e7a3b3eb055b6ad (127.0.97.3:46625): tablet a49accef89094dd79d000cfddd400e13 (table test-table [id=2ba55858041f45f6874f85f53fe3b2f6]) successfully deleted
I20260929 20:46:46.050033 2123 catalog_manager.cc:4841] TS 07c034afa3e44dcdaf52f96e4c4f14dc (127.0.97.2:45057): tablet a49accef89094dd79d000cfddd400e13 (table test-table [id=2ba55858041f45f6874f85f53fe3b2f6]) successfully deleted
I20260929 20:46:46.158934 2649 raft_consensus.cc:482] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:46:46.159157 2649 raft_consensus.cc:504] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:46.159910 2649 leader_election.cc:288] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07c034afa3e44dcdaf52f96e4c4f14dc (127.0.97.2:45057), ac1a30935b2647609e7a3b3eb055b6ad (127.0.97.3:46625)
I20260929 20:46:46.160498 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a89b147e6c43c898db2f126cdd6a5a" candidate_uuid: "3ed8101318574e84b35484a916fa5515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" is_pre_election: true
I20260929 20:46:46.160601 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a89b147e6c43c898db2f126cdd6a5a" candidate_uuid: "3ed8101318574e84b35484a916fa5515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" is_pre_election: true
I20260929 20:46:46.160750 2438 raft_consensus.cc:2454] T 07a89b147e6c43c898db2f126cdd6a5a P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ed8101318574e84b35484a916fa5515 in term 0.
I20260929 20:46:46.160861 2576 raft_consensus.cc:2454] T 07a89b147e6c43c898db2f126cdd6a5a P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ed8101318574e84b35484a916fa5515 in term 0.
I20260929 20:46:46.161310 2224 leader_election.cc:302] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed8101318574e84b35484a916fa5515, ac1a30935b2647609e7a3b3eb055b6ad; no voters:
I20260929 20:46:46.161587 2649 raft_consensus.cc:2781] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:46:46.161710 2649 raft_consensus.cc:482] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:46:46.161805 2649 raft_consensus.cc:3037] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:46.164753 2649 raft_consensus.cc:504] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:46.165771 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a89b147e6c43c898db2f126cdd6a5a" candidate_uuid: "3ed8101318574e84b35484a916fa5515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac1a30935b2647609e7a3b3eb055b6ad"
I20260929 20:46:46.165786 2438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a89b147e6c43c898db2f126cdd6a5a" candidate_uuid: "3ed8101318574e84b35484a916fa5515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc"
I20260929 20:46:46.165956 2576 raft_consensus.cc:3037] T 07a89b147e6c43c898db2f126cdd6a5a P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:46.165973 2438 raft_consensus.cc:3037] T 07a89b147e6c43c898db2f126cdd6a5a P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:46.168968 2438 raft_consensus.cc:2454] T 07a89b147e6c43c898db2f126cdd6a5a P 07c034afa3e44dcdaf52f96e4c4f14dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed8101318574e84b35484a916fa5515 in term 1.
I20260929 20:46:46.169446 2226 leader_election.cc:302] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c034afa3e44dcdaf52f96e4c4f14dc, 3ed8101318574e84b35484a916fa5515; no voters:
I20260929 20:46:46.169057 2576 raft_consensus.cc:2454] T 07a89b147e6c43c898db2f126cdd6a5a P ac1a30935b2647609e7a3b3eb055b6ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ed8101318574e84b35484a916fa5515 in term 1.
I20260929 20:46:46.169962 2652 raft_consensus.cc:2781] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:46:46.170051 2649 leader_election.cc:288] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515 [CANDIDATE]: Term 1 election: Requested vote from peers 07c034afa3e44dcdaf52f96e4c4f14dc (127.0.97.2:45057), ac1a30935b2647609e7a3b3eb055b6ad (127.0.97.3:46625)
I20260929 20:46:46.170154 2652 raft_consensus.cc:686] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515 [term 1 LEADER]: Becoming Leader. State: Replica: 3ed8101318574e84b35484a916fa5515, State: Running, Role: LEADER
I20260929 20:46:46.170491 2652 consensus_queue.cc:237] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:46.174286 2137 catalog_manager.cc:5495] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ed8101318574e84b35484a916fa5515 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "3ed8101318574e84b35484a916fa5515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } health_report { overall_health: UNKNOWN } } }
I20260929 20:46:46.200331 2576 raft_consensus.cc:1261] T 07a89b147e6c43c898db2f126cdd6a5a P ac1a30935b2647609e7a3b3eb055b6ad [term 1 FOLLOWER]: Refusing update from remote peer 3ed8101318574e84b35484a916fa5515: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:46:46.201125 2686 consensus_queue.cc:1035] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:46:46.203596 2438 raft_consensus.cc:1261] T 07a89b147e6c43c898db2f126cdd6a5a P 07c034afa3e44dcdaf52f96e4c4f14dc [term 1 FOLLOWER]: Refusing update from remote peer 3ed8101318574e84b35484a916fa5515: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:46:46.204128 2686 consensus_queue.cc:1035] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:46:46.215426 2670 mvcc.cc:205] Tried to move back new op lower bound from 7334767846190923776 to 7334767846077177856. Current Snapshot: MvccSnapshot[applied={T|T < 7334767846190923776}]
I20260929 20:46:46.215603 2671 mvcc.cc:205] Tried to move back new op lower bound from 7334767846190923776 to 7334767846077177856. Current Snapshot: MvccSnapshot[applied={T|T < 7334767846190923776}]
W20260929 20:46:46.242714 2671 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 20:46:47.834095 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:46:47.834364 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:46:47.839121 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052: Sending DeleteTablet for 3 replicas of tablet 07a89b147e6c43c898db2f126cdd6a5a
I20260929 20:46:47.839955 2274 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a89b147e6c43c898db2f126cdd6a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:46:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58668
I20260929 20:46:47.840132 2551 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a89b147e6c43c898db2f126cdd6a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:46:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37360
I20260929 20:46:47.840354 2413 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a89b147e6c43c898db2f126cdd6a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:46:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58828
I20260929 20:46:47.840616 2721 tablet_replica.cc:330] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515: stopping tablet replica
I20260929 20:46:47.840838 2721 raft_consensus.cc:2229] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:46:47.841199 2721 raft_consensus.cc:2258] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:46:47.842114 2721 ts_tablet_manager.cc:1905] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:46:47.848170 2721 ts_tablet_manager.cc:1918] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:46:47.848331 2721 log.cc:1196] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/07a89b147e6c43c898db2f126cdd6a5a
I20260929 20:46:47.848769 2721 ts_tablet_manager.cc:1939] T 07a89b147e6c43c898db2f126cdd6a5a P 3ed8101318574e84b35484a916fa5515: Deleting consensus metadata
I20260929 20:46:47.849784 2124 catalog_manager.cc:4841] TS 3ed8101318574e84b35484a916fa5515 (127.0.97.1:44991): tablet 07a89b147e6c43c898db2f126cdd6a5a (table test-table [id=992c3f87bc60492e87221c76e2e03bfb]) successfully deleted
I20260929 20:46:47.850121 2722 tablet_replica.cc:330] T 07a89b147e6c43c898db2f126cdd6a5a P ac1a30935b2647609e7a3b3eb055b6ad: stopping tablet replica
I20260929 20:46:47.850301 2722 raft_consensus.cc:2229] T 07a89b147e6c43c898db2f126cdd6a5a P ac1a30935b2647609e7a3b3eb055b6ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:46:47.850492 2722 raft_consensus.cc:2258] T 07a89b147e6c43c898db2f126cdd6a5a P ac1a30935b2647609e7a3b3eb055b6ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:46:47.852124 2722 ts_tablet_manager.cc:1905] T 07a89b147e6c43c898db2f126cdd6a5a P ac1a30935b2647609e7a3b3eb055b6ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:46:47.856539 2722 ts_tablet_manager.cc:1918] T 07a89b147e6c43c898db2f126cdd6a5a P ac1a30935b2647609e7a3b3eb055b6ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:46:47.856647 2722 log.cc:1196] T 07a89b147e6c43c898db2f126cdd6a5a P ac1a30935b2647609e7a3b3eb055b6ad: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/07a89b147e6c43c898db2f126cdd6a5a
I20260929 20:46:47.857049 2723 tablet_replica.cc:330] T 07a89b147e6c43c898db2f126cdd6a5a P 07c034afa3e44dcdaf52f96e4c4f14dc: stopping tablet replica
I20260929 20:46:47.857050 2722 ts_tablet_manager.cc:1939] T 07a89b147e6c43c898db2f126cdd6a5a P ac1a30935b2647609e7a3b3eb055b6ad: Deleting consensus metadata
I20260929 20:46:47.857285 2723 raft_consensus.cc:2229] T 07a89b147e6c43c898db2f126cdd6a5a P 07c034afa3e44dcdaf52f96e4c4f14dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:46:47.857563 2723 raft_consensus.cc:2258] T 07a89b147e6c43c898db2f126cdd6a5a P 07c034afa3e44dcdaf52f96e4c4f14dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:46:47.858083 2121 catalog_manager.cc:4841] TS ac1a30935b2647609e7a3b3eb055b6ad (127.0.97.3:46625): tablet 07a89b147e6c43c898db2f126cdd6a5a (table test-table [id=992c3f87bc60492e87221c76e2e03bfb]) successfully deleted
I20260929 20:46:47.858517 2723 ts_tablet_manager.cc:1905] T 07a89b147e6c43c898db2f126cdd6a5a P 07c034afa3e44dcdaf52f96e4c4f14dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:46:47.864257 2723 ts_tablet_manager.cc:1918] T 07a89b147e6c43c898db2f126cdd6a5a P 07c034afa3e44dcdaf52f96e4c4f14dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:46:47.864398 2723 log.cc:1196] T 07a89b147e6c43c898db2f126cdd6a5a P 07c034afa3e44dcdaf52f96e4c4f14dc: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/07a89b147e6c43c898db2f126cdd6a5a
I20260929 20:46:47.864746 2723 ts_tablet_manager.cc:1939] T 07a89b147e6c43c898db2f126cdd6a5a P 07c034afa3e44dcdaf52f96e4c4f14dc: Deleting consensus metadata
I20260929 20:46:47.865613 2123 catalog_manager.cc:4841] TS 07c034afa3e44dcdaf52f96e4c4f14dc (127.0.97.2:45057): tablet 07a89b147e6c43c898db2f126cdd6a5a (table test-table [id=992c3f87bc60492e87221c76e2e03bfb]) successfully deleted
Sep 29 20:46:47 dist-test-slave-blv7 krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714807, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:46:47 dist-test-slave-blv7 krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714807, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:47.893842 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:47.879736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36116 (local address 127.0.97.62:36609)
0929 20:46:47.880308 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:47.880328 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:47.880407 (+ 79us) server_negotiation.cc:407] Connection header received
0929 20:46:47.880647 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:47.880652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:47.880724 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:47.880825 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:47.882227 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:47.883110 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:47.884451 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:47.884872 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:47.885031 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:47.888698 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:47.888703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:47.888706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:47.888739 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:47.891517 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:47.892093 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:47.892097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:47.892099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:47.892253 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:47.892583 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:47.892589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:47.892592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:47.892838 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:47.893057 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:47.893322 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 20:46:47.893528 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":252,"threads_started":1}
I20260929 20:46:47.896942 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:46:47.897495 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:46:47.903746 2274 tablet_service.cc:1467] Processing CreateTablet for tablet 64f341e1e228405198db46b25e234268 (DEFAULT_TABLE table=test-table [id=a9aea99e2755429fbccb341b1a883833]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:46:47.903975 2413 tablet_service.cc:1467] Processing CreateTablet for tablet 64f341e1e228405198db46b25e234268 (DEFAULT_TABLE table=test-table [id=a9aea99e2755429fbccb341b1a883833]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:46:47.904058 2551 tablet_service.cc:1467] Processing CreateTablet for tablet 64f341e1e228405198db46b25e234268 (DEFAULT_TABLE table=test-table [id=a9aea99e2755429fbccb341b1a883833]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:46:47.904175 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64f341e1e228405198db46b25e234268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:47.904356 2413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64f341e1e228405198db46b25e234268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:47.904450 2551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64f341e1e228405198db46b25e234268. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:47.908907 2732 tablet_bootstrap.cc:492] T 64f341e1e228405198db46b25e234268 P 3ed8101318574e84b35484a916fa5515: Bootstrap starting.
I20260929 20:46:47.910516 2732 tablet_bootstrap.cc:654] T 64f341e1e228405198db46b25e234268 P 3ed8101318574e84b35484a916fa5515: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:47.910737 2733 tablet_bootstrap.cc:492] T 64f341e1e228405198db46b25e234268 P 07c034afa3e44dcdaf52f96e4c4f14dc: Bootstrap starting.
I20260929 20:46:47.912463 2733 tablet_bootstrap.cc:654] T 64f341e1e228405198db46b25e234268 P 07c034afa3e44dcdaf52f96e4c4f14dc: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:47.918556 2732 tablet_bootstrap.cc:492] T 64f341e1e228405198db46b25e234268 P 3ed8101318574e84b35484a916fa5515: No bootstrap required, opened a new log
I20260929 20:46:47.918771 2732 ts_tablet_manager.cc:1397] T 64f341e1e228405198db46b25e234268 P 3ed8101318574e84b35484a916fa5515: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20260929 20:46:47.919983 2734 tablet_bootstrap.cc:492] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad: Bootstrap starting.
I20260929 20:46:47.919880 2732 raft_consensus.cc:348] T 64f341e1e228405198db46b25e234268 P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:47.920192 2732 raft_consensus.cc:374] T 64f341e1e228405198db46b25e234268 P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:47.920328 2732 raft_consensus.cc:729] T 64f341e1e228405198db46b25e234268 P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed8101318574e84b35484a916fa5515, State: Initialized, Role: FOLLOWER
I20260929 20:46:47.920665 2733 tablet_bootstrap.cc:492] T 64f341e1e228405198db46b25e234268 P 07c034afa3e44dcdaf52f96e4c4f14dc: No bootstrap required, opened a new log
I20260929 20:46:47.920657 2732 consensus_queue.cc:260] T 64f341e1e228405198db46b25e234268 P 3ed8101318574e84b35484a916fa5515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:47.921379 2733 ts_tablet_manager.cc:1397] T 64f341e1e228405198db46b25e234268 P 07c034afa3e44dcdaf52f96e4c4f14dc: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260929 20:46:47.921813 2732 ts_tablet_manager.cc:1428] T 64f341e1e228405198db46b25e234268 P 3ed8101318574e84b35484a916fa5515: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 20:46:47.922171 2733 raft_consensus.cc:348] T 64f341e1e228405198db46b25e234268 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:47.922456 2733 raft_consensus.cc:374] T 64f341e1e228405198db46b25e234268 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:47.922591 2733 raft_consensus.cc:729] T 64f341e1e228405198db46b25e234268 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c034afa3e44dcdaf52f96e4c4f14dc, State: Initialized, Role: FOLLOWER
I20260929 20:46:47.922883 2733 consensus_queue.cc:260] T 64f341e1e228405198db46b25e234268 P 07c034afa3e44dcdaf52f96e4c4f14dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:47.923712 2734 tablet_bootstrap.cc:654] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:47.926409 2733 ts_tablet_manager.cc:1428] T 64f341e1e228405198db46b25e234268 P 07c034afa3e44dcdaf52f96e4c4f14dc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 20:46:47.926968 2734 tablet_bootstrap.cc:492] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad: No bootstrap required, opened a new log
I20260929 20:46:47.927368 2734 ts_tablet_manager.cc:1397] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 20:46:47.928072 2734 raft_consensus.cc:348] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:47.928313 2734 raft_consensus.cc:374] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:47.928411 2734 raft_consensus.cc:729] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1a30935b2647609e7a3b3eb055b6ad, State: Initialized, Role: FOLLOWER
I20260929 20:46:47.928670 2734 consensus_queue.cc:260] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:47.929225 2734 ts_tablet_manager.cc:1428] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 20:46:48.058176 2702 raft_consensus.cc:482] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:46:48.058383 2702 raft_consensus.cc:504] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:48.059109 2702 leader_election.cc:288] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ed8101318574e84b35484a916fa5515 (127.0.97.1:44991), 07c034afa3e44dcdaf52f96e4c4f14dc (127.0.97.2:45057)
I20260929 20:46:48.059672 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f341e1e228405198db46b25e234268" candidate_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" is_pre_election: true
I20260929 20:46:48.059667 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f341e1e228405198db46b25e234268" candidate_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed8101318574e84b35484a916fa5515" is_pre_election: true
I20260929 20:46:48.059911 2294 raft_consensus.cc:2454] T 64f341e1e228405198db46b25e234268 P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac1a30935b2647609e7a3b3eb055b6ad in term 0.
I20260929 20:46:48.059911 2437 raft_consensus.cc:2454] T 64f341e1e228405198db46b25e234268 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac1a30935b2647609e7a3b3eb055b6ad in term 0.
I20260929 20:46:48.060413 2506 leader_election.cc:302] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07c034afa3e44dcdaf52f96e4c4f14dc, 3ed8101318574e84b35484a916fa5515, ac1a30935b2647609e7a3b3eb055b6ad; no voters:
I20260929 20:46:48.060709 2702 raft_consensus.cc:2781] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:46:48.060820 2702 raft_consensus.cc:482] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:46:48.060899 2702 raft_consensus.cc:3037] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:48.063366 2702 raft_consensus.cc:504] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:48.063899 2702 leader_election.cc:288] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad [CANDIDATE]: Term 1 election: Requested vote from peers 3ed8101318574e84b35484a916fa5515 (127.0.97.1:44991), 07c034afa3e44dcdaf52f96e4c4f14dc (127.0.97.2:45057)
I20260929 20:46:48.064360 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f341e1e228405198db46b25e234268" candidate_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc"
I20260929 20:46:48.064374 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f341e1e228405198db46b25e234268" candidate_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed8101318574e84b35484a916fa5515"
I20260929 20:46:48.064541 2437 raft_consensus.cc:3037] T 64f341e1e228405198db46b25e234268 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:48.064558 2294 raft_consensus.cc:3037] T 64f341e1e228405198db46b25e234268 P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:48.066396 2294 raft_consensus.cc:2454] T 64f341e1e228405198db46b25e234268 P 3ed8101318574e84b35484a916fa5515 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac1a30935b2647609e7a3b3eb055b6ad in term 1.
I20260929 20:46:48.066565 2437 raft_consensus.cc:2454] T 64f341e1e228405198db46b25e234268 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac1a30935b2647609e7a3b3eb055b6ad in term 1.
I20260929 20:46:48.066748 2506 leader_election.cc:302] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ed8101318574e84b35484a916fa5515, ac1a30935b2647609e7a3b3eb055b6ad; no voters:
I20260929 20:46:48.067009 2702 raft_consensus.cc:2781] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:46:48.067168 2702 raft_consensus.cc:686] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad [term 1 LEADER]: Becoming Leader. State: Replica: ac1a30935b2647609e7a3b3eb055b6ad, State: Running, Role: LEADER
I20260929 20:46:48.067404 2702 consensus_queue.cc:237] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:48.069762 2137 catalog_manager.cc:5495] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad reported cstate change: term changed from 0 to 1, leader changed from <none> to ac1a30935b2647609e7a3b3eb055b6ad (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } health_report { overall_health: HEALTHY } } }
I20260929 20:46:48.117157 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:48.112105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49682 (local address 127.0.97.3:46625)
0929 20:46:48.112553 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:48.112557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:48.112575 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:46:48.112624 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:48.112627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:48.112684 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:48.112759 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:46:48.113120 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.113790 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.115633 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.116003 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.116099 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:48.116498 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:46:48.116592 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:46:48.116651 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:48.116759 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 20:46:48.116813 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":241,"threads_started":1}
I20260929 20:46:48.129158 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:48.124560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43015 (local address 127.0.97.3:46625)
0929 20:46:48.124718 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:48.124722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:48.125175 (+ 453us) server_negotiation.cc:407] Connection header received
0929 20:46:48.125281 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:48.125286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:48.125368 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:48.125472 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:46:48.125848 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.126515 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.127797 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.128178 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.128268 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:48.128596 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:46:48.128686 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:46:48.128751 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:48.128857 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 20:46:48.128911 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":58}
I20260929 20:46:48.129256 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:48.124466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46625 (local address 127.0.97.3:43015)
0929 20:46:48.125078 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:48.125093 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:48.125191 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:48.125470 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:48.125473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:48.125480 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:46:48.125727 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:48.125734 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.126612 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.126615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:48.127679 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:48.127690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.128299 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.128304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:48.128421 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:48.128451 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:46:48.128767 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:46:48.128775 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 20:46:48.128834 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 20:46:48.128966 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":2,"thread_start_us":349,"threads_started":1}
I20260929 20:46:48.136916 2437 raft_consensus.cc:1261] T 64f341e1e228405198db46b25e234268 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 1 FOLLOWER]: Refusing update from remote peer ac1a30935b2647609e7a3b3eb055b6ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:46:48.137009 2294 raft_consensus.cc:1261] T 64f341e1e228405198db46b25e234268 P 3ed8101318574e84b35484a916fa5515 [term 1 FOLLOWER]: Refusing update from remote peer ac1a30935b2647609e7a3b3eb055b6ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:46:48.137403 2702 consensus_queue.cc:1035] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:46:48.137712 2707 consensus_queue.cc:1035] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:46:48.180745 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:46:48.180974 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:46:48.183624 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052: Sending DeleteTablet for 3 replicas of tablet 64f341e1e228405198db46b25e234268
I20260929 20:46:48.184376 2274 tablet_service.cc:1514] Processing DeleteTablet for tablet 64f341e1e228405198db46b25e234268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:46:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58668
I20260929 20:46:48.184670 2721 tablet_replica.cc:330] T 64f341e1e228405198db46b25e234268 P 3ed8101318574e84b35484a916fa5515: stopping tablet replica
I20260929 20:46:48.184633 2552 tablet_service.cc:1514] Processing DeleteTablet for tablet 64f341e1e228405198db46b25e234268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:46:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37360
I20260929 20:46:48.184866 2722 tablet_replica.cc:330] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad: stopping tablet replica
I20260929 20:46:48.185053 2721 raft_consensus.cc:2229] T 64f341e1e228405198db46b25e234268 P 3ed8101318574e84b35484a916fa5515 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:46:48.185058 2722 raft_consensus.cc:2229] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:46:48.185257 2721 raft_consensus.cc:2258] T 64f341e1e228405198db46b25e234268 P 3ed8101318574e84b35484a916fa5515 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:46:48.185464 2722 raft_consensus.cc:2258] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:46:48.186357 2721 ts_tablet_manager.cc:1905] T 64f341e1e228405198db46b25e234268 P 3ed8101318574e84b35484a916fa5515: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:46:48.186642 2722 ts_tablet_manager.cc:1905] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:46:48.188813 2413 tablet_service.cc:1514] Processing DeleteTablet for tablet 64f341e1e228405198db46b25e234268 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:46:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58828
I20260929 20:46:48.189074 2723 tablet_replica.cc:330] T 64f341e1e228405198db46b25e234268 P 07c034afa3e44dcdaf52f96e4c4f14dc: stopping tablet replica
I20260929 20:46:48.189241 2723 raft_consensus.cc:2229] T 64f341e1e228405198db46b25e234268 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:46:48.189070 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 20:46:48.189400 2723 raft_consensus.cc:2258] T 64f341e1e228405198db46b25e234268 P 07c034afa3e44dcdaf52f96e4c4f14dc [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 20:46:48.189668 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:46:48.190570 2723 ts_tablet_manager.cc:1905] T 64f341e1e228405198db46b25e234268 P 07c034afa3e44dcdaf52f96e4c4f14dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:46:48.192636 2721 ts_tablet_manager.cc:1918] T 64f341e1e228405198db46b25e234268 P 3ed8101318574e84b35484a916fa5515: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:46:48.192822 2721 log.cc:1196] T 64f341e1e228405198db46b25e234268 P 3ed8101318574e84b35484a916fa5515: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/64f341e1e228405198db46b25e234268
I20260929 20:46:48.193277 2721 ts_tablet_manager.cc:1939] T 64f341e1e228405198db46b25e234268 P 3ed8101318574e84b35484a916fa5515: Deleting consensus metadata
I20260929 20:46:48.194252 2722 ts_tablet_manager.cc:1918] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:46:48.194267 2124 catalog_manager.cc:4841] TS 3ed8101318574e84b35484a916fa5515 (127.0.97.1:44991): tablet 64f341e1e228405198db46b25e234268 (table test-table [id=a9aea99e2755429fbccb341b1a883833]) successfully deleted
I20260929 20:46:48.194494 2722 log.cc:1196] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/64f341e1e228405198db46b25e234268
I20260929 20:46:48.194847 2722 ts_tablet_manager.cc:1939] T 64f341e1e228405198db46b25e234268 P ac1a30935b2647609e7a3b3eb055b6ad: Deleting consensus metadata
I20260929 20:46:48.195879 2121 catalog_manager.cc:4841] TS ac1a30935b2647609e7a3b3eb055b6ad (127.0.97.3:46625): tablet 64f341e1e228405198db46b25e234268 (table test-table [id=a9aea99e2755429fbccb341b1a883833]) successfully deleted
I20260929 20:46:48.197197 2723 ts_tablet_manager.cc:1918] T 64f341e1e228405198db46b25e234268 P 07c034afa3e44dcdaf52f96e4c4f14dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:46:48.197335 2723 log.cc:1196] T 64f341e1e228405198db46b25e234268 P 07c034afa3e44dcdaf52f96e4c4f14dc: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/64f341e1e228405198db46b25e234268
I20260929 20:46:48.197686 2723 ts_tablet_manager.cc:1939] T 64f341e1e228405198db46b25e234268 P 07c034afa3e44dcdaf52f96e4c4f14dc: Deleting consensus metadata
I20260929 20:46:48.198611 2123 catalog_manager.cc:4841] TS 07c034afa3e44dcdaf52f96e4c4f14dc (127.0.97.2:45057): tablet 64f341e1e228405198db46b25e234268 (table test-table [id=a9aea99e2755429fbccb341b1a883833]) successfully deleted
I20260929 20:46:48.200153 2274 tablet_service.cc:1467] Processing CreateTablet for tablet aa877173b2f842569c7dc7e37fc114fc (DEFAULT_TABLE table=test-table [id=c37f51c521e44170a9af140529a101ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:46:48.200640 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa877173b2f842569c7dc7e37fc114fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:48.203491 2413 tablet_service.cc:1467] Processing CreateTablet for tablet aa877173b2f842569c7dc7e37fc114fc (DEFAULT_TABLE table=test-table [id=c37f51c521e44170a9af140529a101ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:46:48.203863 2413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa877173b2f842569c7dc7e37fc114fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:48.206805 2732 tablet_bootstrap.cc:492] T aa877173b2f842569c7dc7e37fc114fc P 3ed8101318574e84b35484a916fa5515: Bootstrap starting.
I20260929 20:46:48.208377 2732 tablet_bootstrap.cc:654] T aa877173b2f842569c7dc7e37fc114fc P 3ed8101318574e84b35484a916fa5515: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:48.208550 2733 tablet_bootstrap.cc:492] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc: Bootstrap starting.
I20260929 20:46:48.208879 2552 tablet_service.cc:1467] Processing CreateTablet for tablet aa877173b2f842569c7dc7e37fc114fc (DEFAULT_TABLE table=test-table [id=c37f51c521e44170a9af140529a101ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:46:48.209300 2552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa877173b2f842569c7dc7e37fc114fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:48.210602 2733 tablet_bootstrap.cc:654] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:48.218272 2734 tablet_bootstrap.cc:492] T aa877173b2f842569c7dc7e37fc114fc P ac1a30935b2647609e7a3b3eb055b6ad: Bootstrap starting.
I20260929 20:46:48.219965 2734 tablet_bootstrap.cc:654] T aa877173b2f842569c7dc7e37fc114fc P ac1a30935b2647609e7a3b3eb055b6ad: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:48.221680 2732 tablet_bootstrap.cc:492] T aa877173b2f842569c7dc7e37fc114fc P 3ed8101318574e84b35484a916fa5515: No bootstrap required, opened a new log
I20260929 20:46:48.221863 2732 ts_tablet_manager.cc:1397] T aa877173b2f842569c7dc7e37fc114fc P 3ed8101318574e84b35484a916fa5515: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260929 20:46:48.222993 2732 raft_consensus.cc:348] T aa877173b2f842569c7dc7e37fc114fc P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:48.223264 2732 raft_consensus.cc:374] T aa877173b2f842569c7dc7e37fc114fc P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:48.223290 2734 tablet_bootstrap.cc:492] T aa877173b2f842569c7dc7e37fc114fc P ac1a30935b2647609e7a3b3eb055b6ad: No bootstrap required, opened a new log
I20260929 20:46:48.223335 2732 raft_consensus.cc:729] T aa877173b2f842569c7dc7e37fc114fc P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ed8101318574e84b35484a916fa5515, State: Initialized, Role: FOLLOWER
I20260929 20:46:48.223441 2734 ts_tablet_manager.cc:1397] T aa877173b2f842569c7dc7e37fc114fc P ac1a30935b2647609e7a3b3eb055b6ad: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260929 20:46:48.223663 2733 tablet_bootstrap.cc:492] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc: No bootstrap required, opened a new log
I20260929 20:46:48.223557 2732 consensus_queue.cc:260] T aa877173b2f842569c7dc7e37fc114fc P 3ed8101318574e84b35484a916fa5515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:48.223800 2733 ts_tablet_manager.cc:1397] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260929 20:46:48.224050 2732 ts_tablet_manager.cc:1428] T aa877173b2f842569c7dc7e37fc114fc P 3ed8101318574e84b35484a916fa5515: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 20:46:48.224148 2734 raft_consensus.cc:348] T aa877173b2f842569c7dc7e37fc114fc P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:48.224406 2734 raft_consensus.cc:374] T aa877173b2f842569c7dc7e37fc114fc P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:48.224529 2734 raft_consensus.cc:729] T aa877173b2f842569c7dc7e37fc114fc P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1a30935b2647609e7a3b3eb055b6ad, State: Initialized, Role: FOLLOWER
I20260929 20:46:48.224475 2733 raft_consensus.cc:348] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:48.224692 2733 raft_consensus.cc:374] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:48.224763 2733 raft_consensus.cc:729] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c034afa3e44dcdaf52f96e4c4f14dc, State: Initialized, Role: FOLLOWER
I20260929 20:46:48.224865 2734 consensus_queue.cc:260] T aa877173b2f842569c7dc7e37fc114fc P ac1a30935b2647609e7a3b3eb055b6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:48.224964 2733 consensus_queue.cc:260] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:48.225466 2733 ts_tablet_manager.cc:1428] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 20:46:48.226111 2734 ts_tablet_manager.cc:1428] T aa877173b2f842569c7dc7e37fc114fc P ac1a30935b2647609e7a3b3eb055b6ad: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 20:46:48.541664 2737 raft_consensus.cc:482] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:46:48.541962 2737 raft_consensus.cc:504] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:48.543592 2737 leader_election.cc:288] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ed8101318574e84b35484a916fa5515 (127.0.97.1:44991), ac1a30935b2647609e7a3b3eb055b6ad (127.0.97.3:46625)
I20260929 20:46:48.550423 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:48.544079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33257 (local address 127.0.97.1:44991)
0929 20:46:48.544789 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:48.544794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:48.544811 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:46:48.544877 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:48.544882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:48.544952 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:48.545042 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:46:48.545768 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.546758 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.548775 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.549508 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.549618 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:48.549731 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:48.550010 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 20:46:48.550129 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"spinlock_wait_cycles":6016,"thread_start_us":566,"threads_started":1}
I20260929 20:46:48.550465 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:48.543924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44991 (local address 127.0.97.2:33257)
0929 20:46:48.544547 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:48.544571 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:48.544714 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:48.545173 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:48.545179 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:48.545213 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:46:48.545554 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:48.545566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.546941 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.546947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:48.548626 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:48.548637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.549697 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.549703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:48.549845 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:48.549877 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:46:48.549969 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 20:46:48.550100 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":1,"spinlock_wait_cycles":21760,"thread_start_us":340,"threads_started":1}
I20260929 20:46:48.551360 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa877173b2f842569c7dc7e37fc114fc" candidate_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed8101318574e84b35484a916fa5515" is_pre_election: true
I20260929 20:46:48.551571 2293 raft_consensus.cc:2454] T aa877173b2f842569c7dc7e37fc114fc P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07c034afa3e44dcdaf52f96e4c4f14dc in term 0.
I20260929 20:46:48.551955 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:48.544112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44655 (local address 127.0.97.3:46625)
0929 20:46:48.544419 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:48.544428 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:48.545286 (+ 858us) server_negotiation.cc:407] Connection header received
0929 20:46:48.545477 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:48.545481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:48.545535 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:48.545604 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:46:48.546112 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.547691 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.549873 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.551050 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.551163 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:48.551245 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:48.551557 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 20:46:48.551682 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260929 20:46:48.551972 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:48.543932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46625 (local address 127.0.97.2:44655)
0929 20:46:48.545200 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:48.545213 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:48.545349 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:48.545633 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:48.545638 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:48.545659 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:46:48.545971 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:48.545980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.547834 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.547839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:48.549737 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:48.549750 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.551220 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.551226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:48.551386 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:48.551419 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:46:48.551507 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 20:46:48.551623 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1141,"mutex_wait_us":512,"spinlock_wait_cycles":896,"thread_start_us":468,"threads_started":1}
I20260929 20:46:48.552210 2367 leader_election.cc:302] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c034afa3e44dcdaf52f96e4c4f14dc, 3ed8101318574e84b35484a916fa5515; no voters:
I20260929 20:46:48.552579 2737 raft_consensus.cc:2781] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:46:48.552553 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa877173b2f842569c7dc7e37fc114fc" candidate_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" is_pre_election: true
I20260929 20:46:48.552676 2737 raft_consensus.cc:482] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:46:48.552747 2737 raft_consensus.cc:3037] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:48.552791 2577 raft_consensus.cc:2454] T aa877173b2f842569c7dc7e37fc114fc P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07c034afa3e44dcdaf52f96e4c4f14dc in term 0.
I20260929 20:46:48.554930 2737 raft_consensus.cc:504] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:48.555481 2737 leader_election.cc:288] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc [CANDIDATE]: Term 1 election: Requested vote from peers 3ed8101318574e84b35484a916fa5515 (127.0.97.1:44991), ac1a30935b2647609e7a3b3eb055b6ad (127.0.97.3:46625)
I20260929 20:46:48.555968 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa877173b2f842569c7dc7e37fc114fc" candidate_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac1a30935b2647609e7a3b3eb055b6ad"
I20260929 20:46:48.556013 2293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa877173b2f842569c7dc7e37fc114fc" candidate_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ed8101318574e84b35484a916fa5515"
I20260929 20:46:48.556162 2577 raft_consensus.cc:3037] T aa877173b2f842569c7dc7e37fc114fc P ac1a30935b2647609e7a3b3eb055b6ad [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:48.556180 2293 raft_consensus.cc:3037] T aa877173b2f842569c7dc7e37fc114fc P 3ed8101318574e84b35484a916fa5515 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:48.558051 2293 raft_consensus.cc:2454] T aa877173b2f842569c7dc7e37fc114fc P 3ed8101318574e84b35484a916fa5515 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c034afa3e44dcdaf52f96e4c4f14dc in term 1.
I20260929 20:46:48.558387 2367 leader_election.cc:302] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c034afa3e44dcdaf52f96e4c4f14dc, 3ed8101318574e84b35484a916fa5515; no voters:
I20260929 20:46:48.558620 2737 raft_consensus.cc:2781] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:46:48.559062 2737 raft_consensus.cc:686] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc [term 1 LEADER]: Becoming Leader. State: Replica: 07c034afa3e44dcdaf52f96e4c4f14dc, State: Running, Role: LEADER
I20260929 20:46:48.559437 2737 consensus_queue.cc:237] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } }
I20260929 20:46:48.559787 2577 raft_consensus.cc:2454] T aa877173b2f842569c7dc7e37fc114fc P ac1a30935b2647609e7a3b3eb055b6ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c034afa3e44dcdaf52f96e4c4f14dc in term 1.
I20260929 20:46:48.562762 2135 catalog_manager.cc:5495] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 07c034afa3e44dcdaf52f96e4c4f14dc (127.0.97.2). New cstate: current_term: 1 leader_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07c034afa3e44dcdaf52f96e4c4f14dc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 } health_report { overall_health: UNKNOWN } } }
I20260929 20:46:48.593283 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:48.587870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49782 (local address 127.0.97.2:45057)
0929 20:46:48.588329 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:48.588333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:48.588350 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:46:48.588421 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:48.588424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:48.588497 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:48.588623 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:46:48.588917 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.589631 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.590852 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.591749 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.591847 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:48.592206 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:46:48.592743 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:46:48.592843 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:48.592974 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 20:46:48.593040 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":238,"threads_started":1}
I20260929 20:46:48.606325 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:48.601776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37013 (local address 127.0.97.3:46625)
0929 20:46:48.601937 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:48.601941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:48.602412 (+ 471us) server_negotiation.cc:407] Connection header received
0929 20:46:48.602546 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:48.602549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:48.602610 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:48.602679 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:46:48.603100 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.603761 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.604968 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.605381 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.605464 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:48.605850 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:46:48.605940 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:46:48.605995 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:48.606091 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 20:46:48.606143 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 20:46:48.606421 2759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:48.601649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46625 (local address 127.0.97.2:37013)
0929 20:46:48.602330 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:48.602344 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:48.602447 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:48.602690 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:48.602693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:48.602701 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:46:48.602957 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:48.602965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.603870 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.603874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:48.604864 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:48.604874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.605541 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.605546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:48.605678 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:48.605725 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:46:48.606006 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:46:48.606018 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 20:46:48.606073 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 20:46:48.606162 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":406,"threads_started":1}
I20260929 20:46:48.613251 2293 raft_consensus.cc:1261] T aa877173b2f842569c7dc7e37fc114fc P 3ed8101318574e84b35484a916fa5515 [term 1 FOLLOWER]: Refusing update from remote peer 07c034afa3e44dcdaf52f96e4c4f14dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:46:48.613343 2577 raft_consensus.cc:1261] T aa877173b2f842569c7dc7e37fc114fc P ac1a30935b2647609e7a3b3eb055b6ad [term 1 FOLLOWER]: Refusing update from remote peer 07c034afa3e44dcdaf52f96e4c4f14dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:46:48.613869 2754 consensus_queue.cc:1035] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac1a30935b2647609e7a3b3eb055b6ad" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:46:48.614151 2737 consensus_queue.cc:1035] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ed8101318574e84b35484a916fa5515" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:46:48.636660 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:48.631553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54255 (local address 127.0.97.2:45057)
0929 20:46:48.631698 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:48.631704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:48.632120 (+ 416us) server_negotiation.cc:407] Connection header received
0929 20:46:48.632370 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:48.632375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:48.632448 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:48.632512 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:46:48.632903 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.633612 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.635116 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.635636 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.635732 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:48.636089 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:46:48.636213 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:46:48.636289 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:48.636389 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 20:46:48.636455 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 20:46:48.636723 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:48.631482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45057 (local address 127.0.97.3:54255)
0929 20:46:48.632037 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:48.632056 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:48.632218 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:48.632542 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:48.632547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:48.632559 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:46:48.632770 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:48.632778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.633742 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.633747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:48.634965 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:48.634976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:48.635761 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:48.635765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:48.635875 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:48.635910 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:46:48.636300 (+ 390us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:46:48.636307 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:46:48.636365 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 20:46:48.636463 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":328,"threads_started":1}
W20260929 20:46:48.637611 2703 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 20:46:50.219884 2135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:46:50.220111 2135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:46:50.223714 2135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4df9c243e8cc423da9266e8ec6d17052: Sending DeleteTablet for 3 replicas of tablet aa877173b2f842569c7dc7e37fc114fc
I20260929 20:46:50.224435 2274 tablet_service.cc:1514] Processing DeleteTablet for tablet aa877173b2f842569c7dc7e37fc114fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:46:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58668
I20260929 20:46:50.224454 2413 tablet_service.cc:1514] Processing DeleteTablet for tablet aa877173b2f842569c7dc7e37fc114fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:46:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58828
I20260929 20:46:50.224560 2552 tablet_service.cc:1514] Processing DeleteTablet for tablet aa877173b2f842569c7dc7e37fc114fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:46:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37360
I20260929 20:46:50.225121 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 2208
I20260929 20:46:50.236783 2789 tablet_replica.cc:330] T aa877173b2f842569c7dc7e37fc114fc P 3ed8101318574e84b35484a916fa5515: stopping tablet replica
I20260929 20:46:50.237002 2791 tablet_replica.cc:330] T aa877173b2f842569c7dc7e37fc114fc P ac1a30935b2647609e7a3b3eb055b6ad: stopping tablet replica
I20260929 20:46:50.237139 2789 raft_consensus.cc:2229] T aa877173b2f842569c7dc7e37fc114fc P 3ed8101318574e84b35484a916fa5515 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:46:50.237327 2791 raft_consensus.cc:2229] T aa877173b2f842569c7dc7e37fc114fc P ac1a30935b2647609e7a3b3eb055b6ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:46:50.237406 2789 raft_consensus.cc:2258] T aa877173b2f842569c7dc7e37fc114fc P 3ed8101318574e84b35484a916fa5515 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:46:50.237644 2791 raft_consensus.cc:2258] T aa877173b2f842569c7dc7e37fc114fc P ac1a30935b2647609e7a3b3eb055b6ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:46:50.238535 2789 ts_tablet_manager.cc:1905] T aa877173b2f842569c7dc7e37fc114fc P 3ed8101318574e84b35484a916fa5515: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:46:50.239077 2791 ts_tablet_manager.cc:1905] T aa877173b2f842569c7dc7e37fc114fc P ac1a30935b2647609e7a3b3eb055b6ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:46:50.245074 2791 ts_tablet_manager.cc:1918] T aa877173b2f842569c7dc7e37fc114fc P ac1a30935b2647609e7a3b3eb055b6ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:46:50.245093 2789 ts_tablet_manager.cc:1918] T aa877173b2f842569c7dc7e37fc114fc P 3ed8101318574e84b35484a916fa5515: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:46:50.245229 2789 log.cc:1196] T aa877173b2f842569c7dc7e37fc114fc P 3ed8101318574e84b35484a916fa5515: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/aa877173b2f842569c7dc7e37fc114fc
I20260929 20:46:50.245301 2791 log.cc:1196] T aa877173b2f842569c7dc7e37fc114fc P ac1a30935b2647609e7a3b3eb055b6ad: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/aa877173b2f842569c7dc7e37fc114fc
I20260929 20:46:50.245852 2790 tablet_replica.cc:330] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc: stopping tablet replica
I20260929 20:46:50.246037 2790 raft_consensus.cc:2229] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:46:50.246191 2791 ts_tablet_manager.cc:1939] T aa877173b2f842569c7dc7e37fc114fc P ac1a30935b2647609e7a3b3eb055b6ad: Deleting consensus metadata
I20260929 20:46:50.246233 2789 ts_tablet_manager.cc:1939] T aa877173b2f842569c7dc7e37fc114fc P 3ed8101318574e84b35484a916fa5515: Deleting consensus metadata
I20260929 20:46:50.246515 2790 raft_consensus.cc:2258] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:46:50.247378 2121 catalog_manager.cc:4841] TS ac1a30935b2647609e7a3b3eb055b6ad (127.0.97.3:46625): tablet aa877173b2f842569c7dc7e37fc114fc (table test-table [id=c37f51c521e44170a9af140529a101ee]) successfully deleted
I20260929 20:46:50.247411 2790 ts_tablet_manager.cc:1905] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:46:50.247378 2124 catalog_manager.cc:4841] TS 3ed8101318574e84b35484a916fa5515 (127.0.97.1:44991): tablet aa877173b2f842569c7dc7e37fc114fc (table test-table [id=c37f51c521e44170a9af140529a101ee]) successfully deleted
Sep 29 20:46:50 dist-test-slave-blv7 krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714807, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:46:50.251989 2790 ts_tablet_manager.cc:1918] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:46:50.252105 2790 log.cc:1196] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/aa877173b2f842569c7dc7e37fc114fc
I20260929 20:46:50.252450 2790 ts_tablet_manager.cc:1939] T aa877173b2f842569c7dc7e37fc114fc P 07c034afa3e44dcdaf52f96e4c4f14dc: Deleting consensus metadata
I20260929 20:46:50.253439 2123 catalog_manager.cc:4841] TS 07c034afa3e44dcdaf52f96e4c4f14dc (127.0.97.2:45057): tablet aa877173b2f842569c7dc7e37fc114fc (table test-table [id=c37f51c521e44170a9af140529a101ee]) successfully deleted
I20260929 20:46:50.258690 2793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:50.227222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38928 (local address 127.0.97.1:44991)
0929 20:46:50.237250 (+ 10028us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:50.237257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:50.237289 (+ 32us) server_negotiation.cc:407] Connection header received
0929 20:46:50.237356 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:50.237361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:50.237447 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:50.239342 (+ 1895us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:50.239549 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:50.240929 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:50.242999 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:50.243798 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:50.245013 (+ 1215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:50.252983 (+ 7970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:50.252999 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:50.253015 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:50.253049 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:50.256037 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:50.256720 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:50.256727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:50.256729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:50.256866 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:50.257230 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:50.257234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:50.257236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:50.257583 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:50.257846 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:50.258144 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 20:46:50.258353 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9872,"thread_start_us":9707,"threads_started":1}
W20260929 20:46:50.259356 2339 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38928
W20260929 20:46:50.259918 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 2208: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:46:50.260116 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 2208
I20260929 20:46:50.278164 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 2348
Sep 29 20:46:50 dist-test-slave-blv7 krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714807, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:46:50.291896 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:50.278825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49784 (local address 127.0.97.2:45057)
0929 20:46:50.279309 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:50.279313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:50.279329 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:46:50.279380 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:50.279383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:50.279437 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:50.279529 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:50.280643 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:50.281316 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:50.282485 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:50.282861 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:50.282950 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:50.286949 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:50.286963 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:50.286979 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:50.287040 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:50.289608 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:50.290204 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:50.290210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:50.290213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:50.290332 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:50.290596 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:50.290599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:50.290601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:50.290927 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:50.291194 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:50.291417 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 20:46:50.291578 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":268,"threads_started":1}
W20260929 20:46:50.292517 2478 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49784
W20260929 20:46:50.293113 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 2348: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:46:50.293234 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 2348
I20260929 20:46:50.310082 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 2487
Sep 29 20:46:50 dist-test-slave-blv7 krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714807, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:46:50.324080 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:50.310761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49688 (local address 127.0.97.3:46625)
0929 20:46:50.311274 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:50.311279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:50.311294 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:46:50.311345 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:50.311348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:50.311410 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:50.311492 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:50.312690 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:50.313415 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:50.314577 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:50.314944 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:50.315069 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:50.318961 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:50.318975 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:50.318991 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:50.319047 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:50.321482 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:50.322116 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:50.322124 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:50.322126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:50.322287 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:50.322669 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:50.322673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:50.322675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:50.323099 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:50.323343 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:50.323593 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 20:46:50.323775 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":275,"threads_started":1}
W20260929 20:46:50.324728 2617 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49688
W20260929 20:46:50.325260 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 2487: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:46:50.325410 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 2487
I20260929 20:46:50.344662 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 2105
I20260929 20:46:50.345731 2177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:46:50.461263 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 2105
2026-09-29T20:46:50Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8202 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:46:50 dist-test-slave-blv7 krb5kdc[2799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:46:50 dist-test-slave-blv7 krb5kdc[2799](info): set up 2 sockets
Sep 29 20:46:50 dist-test-slave-blv7 krb5kdc[2799](info): commencing operation
krb5kdc: starting...
W20260929 20:46:51.451161 388 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:46:51 dist-test-slave-blv7 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:46:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:46:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:46:51.615209 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44589
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:51.909816 2812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:51.910094 2812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:51.910218 2812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:51.917582 2812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:46:51.917695 2812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:51.917774 2812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:51.917835 2812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:46:51.917871 2812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:46:51.927559 2812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:51.928804 2812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:51.930012 2812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:51.940953 2817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:51.941033 2818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:51.941193 2812 server_base.cc:1034] running on GCE node
W20260929 20:46:51.940999 2820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:51.942106 2812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:51.943346 2812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:51.944523 2812 hybrid_clock.cc:648] HybridClock initialized: now 1790714811944467 us; error 43 us; skew 500 ppm
I20260929 20:46:51.944783 2812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:51 dist-test-slave-blv7 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:51.947592 2812 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:46:51.948885 2812 webserver.cc:460] Webserver started at http://127.0.97.62:39349/ using document root <none> and password file <none>
I20260929 20:46:51.949577 2812 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:51.949676 2812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:51.949959 2812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:51.952623 2812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "fdd3cb5230b445d28a2ebd1bbc4952b9"
format_stamp: "Formatted at 2026-09-29 20:46:51 on dist-test-slave-blv7"
server_key: "4ecde343d067acb3a4caed637605b271"
server_key_iv: "2ca4d482ff4587cf98bd63beaa207109"
server_key_version: "encryptionkey@0"
I20260929 20:46:51.953321 2812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "fdd3cb5230b445d28a2ebd1bbc4952b9"
format_stamp: "Formatted at 2026-09-29 20:46:51 on dist-test-slave-blv7"
server_key: "4ecde343d067acb3a4caed637605b271"
server_key_iv: "2ca4d482ff4587cf98bd63beaa207109"
server_key_version: "encryptionkey@0"
I20260929 20:46:51.957885 2812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 20:46:51.960901 2827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:51.962064 2812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 20:46:51.962206 2812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "fdd3cb5230b445d28a2ebd1bbc4952b9"
format_stamp: "Formatted at 2026-09-29 20:46:51 on dist-test-slave-blv7"
server_key: "4ecde343d067acb3a4caed637605b271"
server_key_iv: "2ca4d482ff4587cf98bd63beaa207109"
server_key_version: "encryptionkey@0"
I20260929 20:46:51.962360 2812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:51.988493 2812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:51.992456 2812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:51.992704 2812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:52.009168 2812 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44589
I20260929 20:46:52.009191 2878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44589 every 8 connection(s)
I20260929 20:46:52.010468 2812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:46:52.014209 2879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:52.019448 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 2812
I20260929 20:46:52.019640 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:46:52.020188 388 external_mini_cluster.cc:1428] Setting key 64e7c969fa4d86998ee0c7495c2f985b
I20260929 20:46:52.024241 2879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9: Bootstrap starting.
I20260929 20:46:52.027616 2879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:52.028653 2879 log.cc:824] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9: Log is configured to *not* fsync() on all Append() calls
I20260929 20:46:52.031147 2879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9: No bootstrap required, opened a new log
Sep 29 20:46:52 dist-test-slave-blv7 krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:52.038319 2879 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd3cb5230b445d28a2ebd1bbc4952b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44589 } }
I20260929 20:46:52.038636 2879 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:52.038775 2879 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd3cb5230b445d28a2ebd1bbc4952b9, State: Initialized, Role: FOLLOWER
I20260929 20:46:52.039410 2879 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd3cb5230b445d28a2ebd1bbc4952b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44589 } }
I20260929 20:46:52.039690 2879 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:46:52.039813 2879 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:46:52.039978 2879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:52.040967 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:52.022069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51142 (local address 127.0.97.62:44589)
0929 20:46:52.022830 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:52.022842 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:52.022880 (+ 38us) server_negotiation.cc:407] Connection header received
0929 20:46:52.024531 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:52.024553 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:52.024920 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:52.025217 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:52.026881 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:52.028209 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:52.029705 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:52.030326 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:52.030501 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:52.034007 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:52.034014 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:52.034028 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:52.034062 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:52.036964 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:52.037528 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:52.037534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:52.037540 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:52.037699 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:52.038044 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:52.038048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:52.038050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:52.038513 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:52.038781 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:52.039223 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 20:46:52.039619 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":358,"threads_started":1}
I20260929 20:46:52.042874 2879 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd3cb5230b445d28a2ebd1bbc4952b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44589 } }
I20260929 20:46:52.043344 2879 leader_election.cc:302] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdd3cb5230b445d28a2ebd1bbc4952b9; no voters:
I20260929 20:46:52.043938 2879 leader_election.cc:288] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:46:52.044067 2884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:46:52.044636 2884 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9 [term 1 LEADER]: Becoming Leader. State: Replica: fdd3cb5230b445d28a2ebd1bbc4952b9, State: Running, Role: LEADER
I20260929 20:46:52.045130 2884 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd3cb5230b445d28a2ebd1bbc4952b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44589 } }
I20260929 20:46:52.045667 2879 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:46:52.047482 2886 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdd3cb5230b445d28a2ebd1bbc4952b9. Latest consensus state: current_term: 1 leader_uuid: "fdd3cb5230b445d28a2ebd1bbc4952b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd3cb5230b445d28a2ebd1bbc4952b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44589 } } }
I20260929 20:46:52.047855 2886 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9 [sys.catalog]: This master's current role is: LEADER
I20260929 20:46:52.047668 2885 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdd3cb5230b445d28a2ebd1bbc4952b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd3cb5230b445d28a2ebd1bbc4952b9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44589 } } }
I20260929 20:46:52.047959 2885 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9 [sys.catalog]: This master's current role is: LEADER
I20260929 20:46:52.049561 2890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:46:52.054507 2890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:46:52.062299 2890 catalog_manager.cc:1347] Generated new cluster ID: b78ee905fce54dbbb9797d1585b561c8
I20260929 20:46:52.062402 2890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:46:52.082052 2890 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:46:52.083837 2890 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:46:52.092777 2890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdd3cb5230b445d28a2ebd1bbc4952b9: Generated new TSK 0
I20260929 20:46:52.093534 2890 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:46:52.132874 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44589
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:52.432914 2905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:52.433187 2905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:52.433313 2905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:52.440207 2905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:52.440318 2905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:52.440464 2905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:46:52.450912 2905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:52.452149 2905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:52.453323 2905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:52.465368 2910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:52.465358 2911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:52.465534 2913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:52.466115 2905 server_base.cc:1034] running on GCE node
I20260929 20:46:52.466672 2905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:52.467576 2905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:52.468765 2905 hybrid_clock.cc:648] HybridClock initialized: now 1790714812468708 us; error 50 us; skew 500 ppm
I20260929 20:46:52.469043 2905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:52 dist-test-slave-blv7 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:52.471846 2905 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:46:52.473165 2905 webserver.cc:460] Webserver started at http://127.0.97.1:35563/ using document root <none> and password file <none>
I20260929 20:46:52.473760 2905 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:52.473865 2905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:52.474128 2905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:52.476771 2905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "fef58b76f358467ab98e3538e468ccce"
format_stamp: "Formatted at 2026-09-29 20:46:52 on dist-test-slave-blv7"
server_key: "4b3a22949dab2e01ed48280182427395"
server_key_iv: "ba5537dc016decca0d5765623b190a79"
server_key_version: "encryptionkey@0"
I20260929 20:46:52.477474 2905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fef58b76f358467ab98e3538e468ccce"
format_stamp: "Formatted at 2026-09-29 20:46:52 on dist-test-slave-blv7"
server_key: "4b3a22949dab2e01ed48280182427395"
server_key_iv: "ba5537dc016decca0d5765623b190a79"
server_key_version: "encryptionkey@0"
I20260929 20:46:52.482158 2905 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260929 20:46:52.485292 2920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:52.486506 2905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 20:46:52.486639 2905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "fef58b76f358467ab98e3538e468ccce"
format_stamp: "Formatted at 2026-09-29 20:46:52 on dist-test-slave-blv7"
server_key: "4b3a22949dab2e01ed48280182427395"
server_key_iv: "ba5537dc016decca0d5765623b190a79"
server_key_version: "encryptionkey@0"
I20260929 20:46:52.486795 2905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:52.506726 2905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:52.510500 2905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:52.510731 2905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:52.511615 2905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:46:52.512794 2905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:46:52.512863 2905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:52.512986 2905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:46:52.513037 2905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:52.542171 2905 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44201
I20260929 20:46:52.542219 3032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44201 every 8 connection(s)
I20260929 20:46:52.543381 2905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:46:52.549103 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 2905
I20260929 20:46:52.549299 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:46:52.550033 388 external_mini_cluster.cc:1428] Setting key 611008beb781042bc762022ba86859bf
Sep 29 20:46:52 dist-test-slave-blv7 krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:52.564154 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:52.545678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53855 (local address 127.0.97.62:44589)
0929 20:46:52.546104 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:52.546110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:52.547162 (+ 1052us) server_negotiation.cc:407] Connection header received
0929 20:46:52.549040 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:52.549044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:52.549109 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:52.549221 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:52.551209 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:52.552135 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:52.554323 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:52.554718 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:52.554806 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:52.558531 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:52.558538 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:52.558543 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:52.558578 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:52.561358 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:52.561974 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:52.561979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:52.561983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:52.562106 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:52.562465 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:52.562470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:52.562473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:52.562780 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:52.562911 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:52.563599 (+ 688us) server_negotiation.cc:299] Negotiation successful
0929 20:46:52.563799 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":197,"threads_started":1}
I20260929 20:46:52.564945 3036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:52.546292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44589 (local address 127.0.97.1:53855)
0929 20:46:52.546999 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:52.547062 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:52.548767 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:52.549545 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:52.549557 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:52.550236 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:52.550979 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:52.551004 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:52.552285 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:52.552291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:52.554188 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:52.554202 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:52.554915 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:52.554931 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:52.555207 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:52.555864 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:52.555892 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:52.558312 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:52.561480 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:52.561486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:52.561499 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:52.561867 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:52.562199 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:52.562203 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:52.562205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:52.562370 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:52.562880 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:52.562885 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:52.563253 (+ 368us) client_negotiation.cc:769] Sending connection context
0929 20:46:52.563462 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 20:46:52.563817 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":284,"threads_started":1}
I20260929 20:46:52.566378 3033 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44589
I20260929 20:46:52.566702 3033 heartbeater.cc:464] Registering TS with master...
I20260929 20:46:52.567620 3033 heartbeater.cc:511] Master 127.0.97.62:44589 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:46:52.569710 2844 ts_manager.cc:194] Registered new tserver with Master: fef58b76f358467ab98e3538e468ccce (127.0.97.1:44201)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 20:46:52.571422 2844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:46:52.590483 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44589
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:52.888512 3039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:52.888767 3039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:52.888909 3039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:52.895861 3039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:52.895977 3039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:52.896123 3039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:46:52.905136 3039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:52.906294 3039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:52.907558 3039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:52.919601 3045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:52.919683 3044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:52.919576 3047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:52.920248 3039 server_base.cc:1034] running on GCE node
I20260929 20:46:52.920809 3039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:52.921663 3039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:52.922852 3039 hybrid_clock.cc:648] HybridClock initialized: now 1790714812922817 us; error 29 us; skew 500 ppm
I20260929 20:46:52.923187 3039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:52 dist-test-slave-blv7 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:52.926013 3039 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:46:52.927275 3039 webserver.cc:460] Webserver started at http://127.0.97.2:43207/ using document root <none> and password file <none>
I20260929 20:46:52.927884 3039 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:52.927976 3039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:52.928206 3039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:52.930860 3039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "3ebdba8c94124cb4baa9c5a40cff7ef0"
format_stamp: "Formatted at 2026-09-29 20:46:52 on dist-test-slave-blv7"
server_key: "b2198424bc77a1605798a1240e74f469"
server_key_iv: "7dacbccbf40572f92d89973f5e241213"
server_key_version: "encryptionkey@0"
I20260929 20:46:52.931589 3039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3ebdba8c94124cb4baa9c5a40cff7ef0"
format_stamp: "Formatted at 2026-09-29 20:46:52 on dist-test-slave-blv7"
server_key: "b2198424bc77a1605798a1240e74f469"
server_key_iv: "7dacbccbf40572f92d89973f5e241213"
server_key_version: "encryptionkey@0"
I20260929 20:46:52.936575 3039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 20:46:52.939963 3054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:52.941221 3039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:46:52.941339 3039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "3ebdba8c94124cb4baa9c5a40cff7ef0"
format_stamp: "Formatted at 2026-09-29 20:46:52 on dist-test-slave-blv7"
server_key: "b2198424bc77a1605798a1240e74f469"
server_key_iv: "7dacbccbf40572f92d89973f5e241213"
server_key_version: "encryptionkey@0"
I20260929 20:46:52.941495 3039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:52.961756 3039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:52.965688 3039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:52.965991 3039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:52.966850 3039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:46:52.968070 3039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:46:52.968137 3039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:52.968222 3039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:46:52.968262 3039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:52.995940 3039 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39361
I20260929 20:46:52.995976 3166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39361 every 8 connection(s)
I20260929 20:46:52.997256 3039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:46:53.004663 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 3039
I20260929 20:46:53.004843 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:46:53.005364 388 external_mini_cluster.cc:1428] Setting key 9833ae0e965d8b4a7db28b0e245ede43
Sep 29 20:46:53 dist-test-slave-blv7 krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:53.017068 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:52.999600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45685 (local address 127.0.97.62:44589)
0929 20:46:52.999766 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:52.999770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:53.000881 (+ 1111us) server_negotiation.cc:407] Connection header received
0929 20:46:53.002729 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:53.002735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:53.002816 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:53.002924 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:53.004388 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:53.005473 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:53.006918 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:53.007395 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:53.007504 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:53.011355 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:53.011361 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:53.011366 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:53.011404 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:53.014269 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:53.014957 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:53.014965 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:53.014967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:53.015146 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:53.015524 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:53.015529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:53.015533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:53.015790 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:53.015920 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:53.016520 (+ 600us) server_negotiation.cc:299] Negotiation successful
0929 20:46:53.016732 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:46:53.018100 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:53.000082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44589 (local address 127.0.97.2:45685)
0929 20:46:53.000745 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:53.000781 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:53.002439 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:53.003184 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:53.003195 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:53.003701 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:53.004230 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:53.004246 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:53.005615 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:53.005620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:53.006788 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:53.006799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:53.007601 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:53.007609 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:53.007827 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:53.008434 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:53.008467 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:53.011150 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:53.014392 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:53.014401 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:53.014423 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:53.014806 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:53.015244 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:53.015249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:53.015252 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:53.015426 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:53.015901 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:53.015908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:53.016202 (+ 294us) client_negotiation.cc:769] Sending connection context
0929 20:46:53.016438 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 20:46:53.016928 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":247,"threads_started":1}
I20260929 20:46:53.019476 3167 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44589
I20260929 20:46:53.019745 3167 heartbeater.cc:464] Registering TS with master...
I20260929 20:46:53.020401 3167 heartbeater.cc:511] Master 127.0.97.62:44589 requested a full tablet report, sending...
I20260929 20:46:53.021723 2844 ts_manager.cc:194] Registered new tserver with Master: 3ebdba8c94124cb4baa9c5a40cff7ef0 (127.0.97.2:39361)
I20260929 20:46:53.022501 2844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45685
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:46:53.046226 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44589
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:53.356184 3172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:53.356462 3172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:53.356601 3172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:53.363519 3172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:53.363634 3172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:53.363781 3172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:46:53.372972 3172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:53.374140 3172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:53.375385 3172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:53.387547 3178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:53.387696 3177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:53.387650 3180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:53.388451 3172 server_base.cc:1034] running on GCE node
I20260929 20:46:53.389022 3172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:53.389945 3172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:53.391105 3172 hybrid_clock.cc:648] HybridClock initialized: now 1790714813391054 us; error 46 us; skew 500 ppm
I20260929 20:46:53.391376 3172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:53 dist-test-slave-blv7 krb5kdc[2799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:53.394268 3172 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:46:53.395605 3172 webserver.cc:460] Webserver started at http://127.0.97.3:45473/ using document root <none> and password file <none>
I20260929 20:46:53.396210 3172 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:53.396317 3172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:53.396585 3172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:53.399294 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "904687f1763e4ca581c1a31ba3f2dd56"
format_stamp: "Formatted at 2026-09-29 20:46:53 on dist-test-slave-blv7"
server_key: "95ab02d5b57ab84c0bfb6a46f0ffd3b5"
server_key_iv: "c8952f5a1ee6b40d43b6af22a28218ba"
server_key_version: "encryptionkey@0"
I20260929 20:46:53.399998 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "904687f1763e4ca581c1a31ba3f2dd56"
format_stamp: "Formatted at 2026-09-29 20:46:53 on dist-test-slave-blv7"
server_key: "95ab02d5b57ab84c0bfb6a46f0ffd3b5"
server_key_iv: "c8952f5a1ee6b40d43b6af22a28218ba"
server_key_version: "encryptionkey@0"
I20260929 20:46:53.404644 3172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:46:53.407815 3187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:53.409054 3172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 20:46:53.409193 3172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "904687f1763e4ca581c1a31ba3f2dd56"
format_stamp: "Formatted at 2026-09-29 20:46:53 on dist-test-slave-blv7"
server_key: "95ab02d5b57ab84c0bfb6a46f0ffd3b5"
server_key_iv: "c8952f5a1ee6b40d43b6af22a28218ba"
server_key_version: "encryptionkey@0"
I20260929 20:46:53.409363 3172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:53.425649 3172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:53.429510 3172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:53.429756 3172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:53.430617 3172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:46:53.431845 3172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:46:53.431916 3172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:53.432042 3172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:46:53.432096 3172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:53.460237 3172 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42437
I20260929 20:46:53.460271 3299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42437 every 8 connection(s)
I20260929 20:46:53.461423 3172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:46:53.463678 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 3172
I20260929 20:46:53.463856 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:46:53.464268 388 external_mini_cluster.cc:1428] Setting key bf8128ff9f50926621d1406cdad5f99f
Sep 29 20:46:53 dist-test-slave-blv7 krb5kdc[2799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:53.479868 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:53.463786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53515 (local address 127.0.97.62:44589)
0929 20:46:53.463982 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:53.463988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:53.465197 (+ 1209us) server_negotiation.cc:407] Connection header received
0929 20:46:53.467362 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:53.467367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:53.467443 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:53.467531 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:53.468910 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:53.469619 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:53.471122 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:53.471523 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:53.471625 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:53.474653 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:53.474658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:53.474661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:53.474690 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:53.476827 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:53.477522 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:53.477527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:53.477531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:53.477685 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:53.478326 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:53.478329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:53.478331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:53.478659 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:53.478762 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:53.479476 (+ 714us) server_negotiation.cc:299] Negotiation successful
0929 20:46:53.479638 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:46:53.480719 3302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:53.464288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44589 (local address 127.0.97.3:53515)
0929 20:46:53.465053 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:53.465100 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:53.467126 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:53.467757 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:53.467766 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:53.468245 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:53.468762 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:53.468778 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:53.469767 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:53.469772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:53.470959 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:53.470970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:53.471714 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:53.471721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:53.471953 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:53.472503 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:53.472524 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:53.474463 (+ 1939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:53.476941 (+ 2478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:53.476949 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:53.476964 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:53.477423 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:53.477794 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:53.477800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:53.477803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:53.478036 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:53.478772 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:53.478780 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:53.479179 (+ 399us) client_negotiation.cc:769] Sending connection context
0929 20:46:53.479395 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 20:46:53.479739 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":282,"threads_started":1}
I20260929 20:46:53.481803 3300 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44589
I20260929 20:46:53.482060 3300 heartbeater.cc:464] Registering TS with master...
I20260929 20:46:53.482753 3300 heartbeater.cc:511] Master 127.0.97.62:44589 requested a full tablet report, sending...
I20260929 20:46:53.484052 2844 ts_manager.cc:194] Registered new tserver with Master: 904687f1763e4ca581c1a31ba3f2dd56 (127.0.97.3:42437)
I20260929 20:46:53.484805 2844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53515
I20260929 20:46:53.490983 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 20:46:53.516470 3035 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:46:53.514158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51152 (local address 127.0.97.62:44589)
0929 20:46:53.514364 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:53.514372 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:53.514756 (+ 384us) server_negotiation.cc:407] Connection header received
0929 20:46:53.514917 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:53.514922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:53.515001 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:53.515125 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:53.516232 (+ 1107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51152: BlockingRecv error: recv got EOF from 127.0.0.1:51152 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:46:53.519580 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 2905
W20260929 20:46:53.521828 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 2905: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:44201: server requires authentication, but client does not have Kerberos credentials available
I20260929 20:46:53.522012 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 2905
I20260929 20:46:53.534003 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 3039
W20260929 20:46:53.536216 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 3039: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:39361: server requires authentication, but client does not have Kerberos credentials available
I20260929 20:46:53.536370 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 3039
I20260929 20:46:53.548246 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 3172
W20260929 20:46:53.550836 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 3172: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:42437: server requires authentication, but client does not have Kerberos credentials available
I20260929 20:46:53.550956 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 3172
I20260929 20:46:53.562040 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 2812
I20260929 20:46:53.563180 2874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:46:53.659047 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 2812
2026-09-29T20:46:53Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3195 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:46:53 dist-test-slave-blv7 krb5kdc[3317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:46:53 dist-test-slave-blv7 krb5kdc[3317](info): set up 2 sockets
Sep 29 20:46:53 dist-test-slave-blv7 krb5kdc[3317](info): commencing operation
krb5kdc: starting...
W20260929 20:46:54.638907 388 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:46:54 dist-test-slave-blv7 krb5kdc[3317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:46:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:46:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:46:54.800462 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32999
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:55.095577 3330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:55.095830 3330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:55.095947 3330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:55.102828 3330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:46:55.102922 3330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:55.102994 3330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:55.103091 3330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:46:55.103149 3330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:46:55.112386 3330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:55.113600 3330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:55.114827 3330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:55.125353 3335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:55.125435 3338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:55.125449 3336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:55.126510 3330 server_base.cc:1034] running on GCE node
I20260929 20:46:55.127241 3330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:55.128495 3330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:55.129709 3330 hybrid_clock.cc:648] HybridClock initialized: now 1790714815129659 us; error 36 us; skew 500 ppm
I20260929 20:46:55.130036 3330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:55 dist-test-slave-blv7 krb5kdc[3317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:55.132859 3330 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:46:55.134187 3330 webserver.cc:460] Webserver started at http://127.0.97.62:37009/ using document root <none> and password file <none>
I20260929 20:46:55.134783 3330 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:55.134895 3330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:55.135211 3330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:55.137935 3330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "9bdc266b646149e0932e7700544f2a02"
format_stamp: "Formatted at 2026-09-29 20:46:55 on dist-test-slave-blv7"
server_key: "4a0a7763d85e5a1207d19c12ad129395"
server_key_iv: "d74547e3f5a617274af8c1b7257c7c2f"
server_key_version: "encryptionkey@0"
I20260929 20:46:55.138649 3330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "9bdc266b646149e0932e7700544f2a02"
format_stamp: "Formatted at 2026-09-29 20:46:55 on dist-test-slave-blv7"
server_key: "4a0a7763d85e5a1207d19c12ad129395"
server_key_iv: "d74547e3f5a617274af8c1b7257c7c2f"
server_key_version: "encryptionkey@0"
I20260929 20:46:55.144120 3330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 20:46:55.147892 3345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:55.149215 3330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:46:55.149360 3330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "9bdc266b646149e0932e7700544f2a02"
format_stamp: "Formatted at 2026-09-29 20:46:55 on dist-test-slave-blv7"
server_key: "4a0a7763d85e5a1207d19c12ad129395"
server_key_iv: "d74547e3f5a617274af8c1b7257c7c2f"
server_key_version: "encryptionkey@0"
I20260929 20:46:55.149521 3330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:55.178984 3330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:55.183996 3330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:55.184276 3330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:55.202215 3330 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32999
I20260929 20:46:55.202217 3396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32999 every 8 connection(s)
I20260929 20:46:55.203815 3330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:46:55.206494 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 3330
I20260929 20:46:55.206693 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:46:55.207511 388 external_mini_cluster.cc:1428] Setting key 60205d49f27470382dfbb6388738b9bf
I20260929 20:46:55.207937 3397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:46:55.221629 3397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02: Bootstrap starting.
Sep 29 20:46:55 dist-test-slave-blv7 krb5kdc[3317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:55.225314 3397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:55.226563 3397 log.cc:824] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02: Log is configured to *not* fsync() on all Append() calls
I20260929 20:46:55.229566 3397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02: No bootstrap required, opened a new log
I20260929 20:46:55.230966 3400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:55.209274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46332 (local address 127.0.97.62:32999)
0929 20:46:55.210076 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:55.210090 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:55.210137 (+ 47us) server_negotiation.cc:407] Connection header received
0929 20:46:55.212408 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:55.212436 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:55.212938 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:55.213511 (+ 573us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:55.214727 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:55.216730 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:55.217958 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:55.219009 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:55.219212 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:55.222951 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:55.222962 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:55.222985 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:55.223058 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:55.226636 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:55.227429 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:55.227434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:55.227440 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:55.227600 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:55.228011 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:55.228016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:55.228019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:55.228641 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:55.228912 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:55.229288 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 20:46:55.229672 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":345,"threads_started":1}
I20260929 20:46:55.234709 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:46:55.237582 3397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bdc266b646149e0932e7700544f2a02" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32999 } }
I20260929 20:46:55.237892 3397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:55.237982 3397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bdc266b646149e0932e7700544f2a02, State: Initialized, Role: FOLLOWER
I20260929 20:46:55.238535 3397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bdc266b646149e0932e7700544f2a02" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32999 } }
I20260929 20:46:55.238735 3397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:46:55.238847 3397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:46:55.238989 3397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:55.240962 3397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bdc266b646149e0932e7700544f2a02" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32999 } }
I20260929 20:46:55.241405 3397 leader_election.cc:302] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bdc266b646149e0932e7700544f2a02; no voters:
I20260929 20:46:55.241964 3397 leader_election.cc:288] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:46:55.242071 3403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:46:55.243251 3403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02 [term 1 LEADER]: Becoming Leader. State: Replica: 9bdc266b646149e0932e7700544f2a02, State: Running, Role: LEADER
I20260929 20:46:55.243763 3403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bdc266b646149e0932e7700544f2a02" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32999 } }
I20260929 20:46:55.244117 3397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:46:55.246378 3405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bdc266b646149e0932e7700544f2a02. Latest consensus state: current_term: 1 leader_uuid: "9bdc266b646149e0932e7700544f2a02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bdc266b646149e0932e7700544f2a02" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32999 } } }
I20260929 20:46:55.246714 3405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02 [sys.catalog]: This master's current role is: LEADER
I20260929 20:46:55.247264 3404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bdc266b646149e0932e7700544f2a02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bdc266b646149e0932e7700544f2a02" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32999 } } }
I20260929 20:46:55.247586 3404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02 [sys.catalog]: This master's current role is: LEADER
I20260929 20:46:55.250557 3411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:46:55.261926 3411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:46:55.281706 3411 catalog_manager.cc:1347] Generated new cluster ID: f7e0d6f42ea84c8a818132d4978f00e2
I20260929 20:46:55.281822 3411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:46:55.309545 3400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:55.297191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46346 (local address 127.0.97.62:32999)
0929 20:46:55.297366 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:55.297373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:55.297921 (+ 548us) server_negotiation.cc:407] Connection header received
0929 20:46:55.298111 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:55.298117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:55.298192 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:55.298297 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:55.299499 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:55.300558 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:55.302343 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:55.302985 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:55.303114 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:55.307998 (+ 4884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:55.308005 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:55.308010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:46:55.308790 (+ 780us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:55.308839 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:46:55.308869 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:55.308986 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:55.309119 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 20:46:55.309208 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:46:55.316399 3411 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:46:55.318135 3411 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:46:55.325677 3411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bdc266b646149e0932e7700544f2a02: Generated new TSK 0
I20260929 20:46:55.326576 3411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:46:55.334424 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 3330
I20260929 20:46:55.335958 3392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:46:55.425246 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 3330
2026-09-29T20:46:55Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1761 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:46:55 dist-test-slave-blv7 krb5kdc[3432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:46:55 dist-test-slave-blv7 krb5kdc[3432](info): set up 2 sockets
Sep 29 20:46:55 dist-test-slave-blv7 krb5kdc[3432](info): commencing operation
krb5kdc: starting...
W20260929 20:46:56.399706 388 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:46:56 dist-test-slave-blv7 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:46:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:46:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:46:56.571154 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39059
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:56.865257 3445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:56.865509 3445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:56.865763 3445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:56.872802 3445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:46:56.872895 3445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:56.872965 3445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:56.873034 3445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:46:56.873095 3445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:46:56.882393 3445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:56.883607 3445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:56.884795 3445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:56.895344 3450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:56.895334 3453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:56.895350 3451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:56.895911 3445 server_base.cc:1034] running on GCE node
I20260929 20:46:56.896592 3445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:56.897799 3445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:56.898994 3445 hybrid_clock.cc:648] HybridClock initialized: now 1790714816898932 us; error 49 us; skew 500 ppm
I20260929 20:46:56.899312 3445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:56 dist-test-slave-blv7 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:56.902047 3445 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:46:56.903263 3445 webserver.cc:460] Webserver started at http://127.0.97.62:40761/ using document root <none> and password file <none>
I20260929 20:46:56.903851 3445 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:56.903952 3445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:56.904209 3445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:56.906831 3445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "dbf74b7182fa416d90f2854663c8f727"
format_stamp: "Formatted at 2026-09-29 20:46:56 on dist-test-slave-blv7"
server_key: "3767238e4e4a65cedf232d45ae74d5f0"
server_key_iv: "7d88b5ec16a448494565408c538e8680"
server_key_version: "encryptionkey@0"
I20260929 20:46:56.907555 3445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "dbf74b7182fa416d90f2854663c8f727"
format_stamp: "Formatted at 2026-09-29 20:46:56 on dist-test-slave-blv7"
server_key: "3767238e4e4a65cedf232d45ae74d5f0"
server_key_iv: "7d88b5ec16a448494565408c538e8680"
server_key_version: "encryptionkey@0"
I20260929 20:46:56.912335 3445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:46:56.915477 3460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:56.916682 3445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:46:56.916849 3445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "dbf74b7182fa416d90f2854663c8f727"
format_stamp: "Formatted at 2026-09-29 20:46:56 on dist-test-slave-blv7"
server_key: "3767238e4e4a65cedf232d45ae74d5f0"
server_key_iv: "7d88b5ec16a448494565408c538e8680"
server_key_version: "encryptionkey@0"
I20260929 20:46:56.917027 3445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:56.942950 3445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:56.946949 3445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:56.947247 3445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:56.962950 3445 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39059
I20260929 20:46:56.962968 3511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39059 every 8 connection(s)
I20260929 20:46:56.964262 3445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:46:56.964931 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 3445
I20260929 20:46:56.965132 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:46:56.965806 388 external_mini_cluster.cc:1428] Setting key 1d4d09a464604fe4f509076f845effda
I20260929 20:46:56.968319 3512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:46:56 dist-test-slave-blv7 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:56.981055 3512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727: Bootstrap starting.
I20260929 20:46:56.985030 3512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727: Neither blocks nor log segments found. Creating new log.
I20260929 20:46:56.986306 3512 log.cc:824] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727: Log is configured to *not* fsync() on all Append() calls
I20260929 20:46:56.988547 3515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:56.967671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55638 (local address 127.0.97.62:39059)
0929 20:46:56.968414 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:56.968429 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:56.968467 (+ 38us) server_negotiation.cc:407] Connection header received
0929 20:46:56.969958 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:56.969978 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:56.970330 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:56.970639 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:56.972325 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:56.973582 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:56.975429 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:56.976058 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:56.976179 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:56.980698 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:56.980709 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:56.980728 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:56.980780 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:56.984466 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:56.985057 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:56.985066 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:56.985074 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:56.985264 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:56.985616 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:56.985620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:56.985624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:56.986324 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:56.986627 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:56.986969 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 20:46:56.987337 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":297,"threads_started":1}
I20260929 20:46:56.989672 3512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727: No bootstrap required, opened a new log
I20260929 20:46:56.997371 3512 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf74b7182fa416d90f2854663c8f727" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39059 } }
I20260929 20:46:56.997772 3512 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:46:56.997859 3512 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbf74b7182fa416d90f2854663c8f727, State: Initialized, Role: FOLLOWER
I20260929 20:46:56.998340 3512 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf74b7182fa416d90f2854663c8f727" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39059 } }
I20260929 20:46:56.998553 3512 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:46:56.998653 3512 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:46:56.998782 3512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:46:57.001173 3512 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf74b7182fa416d90f2854663c8f727" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39059 } }
I20260929 20:46:57.001636 3512 leader_election.cc:302] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbf74b7182fa416d90f2854663c8f727; no voters:
I20260929 20:46:57.002184 3512 leader_election.cc:288] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:46:57.002295 3517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:46:57.002894 3517 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727 [term 1 LEADER]: Becoming Leader. State: Replica: dbf74b7182fa416d90f2854663c8f727, State: Running, Role: LEADER
I20260929 20:46:57.003479 3517 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf74b7182fa416d90f2854663c8f727" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39059 } }
I20260929 20:46:57.004209 3512 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:46:57.006111 3519 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbf74b7182fa416d90f2854663c8f727. Latest consensus state: current_term: 1 leader_uuid: "dbf74b7182fa416d90f2854663c8f727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf74b7182fa416d90f2854663c8f727" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39059 } } }
I20260929 20:46:57.006140 3518 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbf74b7182fa416d90f2854663c8f727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf74b7182fa416d90f2854663c8f727" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39059 } } }
I20260929 20:46:57.006453 3519 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727 [sys.catalog]: This master's current role is: LEADER
I20260929 20:46:57.006453 3518 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727 [sys.catalog]: This master's current role is: LEADER
I20260929 20:46:57.007402 3522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:46:57.012485 3522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:46:57.020941 3522 catalog_manager.cc:1347] Generated new cluster ID: 56ac7935fe784021bd3324923b16a562
I20260929 20:46:57.021054 3522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:46:57.037940 3522 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:46:57.039199 3522 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:46:57.052055 3522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbf74b7182fa416d90f2854663c8f727: Generated new TSK 0
I20260929 20:46:57.052829 3522 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:46:57.104278 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39059
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:57.422641 3538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:57.422886 3538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:57.423043 3538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:57.430029 3538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:57.430120 3538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:57.430227 3538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:46:57.439388 3538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:57.440529 3538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:57.441811 3538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:57.453946 3544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:57.454002 3543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:57.454641 3546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:57.454619 3538 server_base.cc:1034] running on GCE node
I20260929 20:46:57.455334 3538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:57.456281 3538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:57.457497 3538 hybrid_clock.cc:648] HybridClock initialized: now 1790714817457443 us; error 37 us; skew 500 ppm
I20260929 20:46:57.457811 3538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:57 dist-test-slave-blv7 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:57.461004 3538 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:46:57.462347 3538 webserver.cc:460] Webserver started at http://127.0.97.1:39717/ using document root <none> and password file <none>
I20260929 20:46:57.463052 3538 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:57.463182 3538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:57.463469 3538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:57.466322 3538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "a3e568cfaaa84e02ae618207bf2e3fe9"
format_stamp: "Formatted at 2026-09-29 20:46:57 on dist-test-slave-blv7"
server_key: "adba186d82cb77e76db51a8289a04a34"
server_key_iv: "7efa7d3b00f840619c2d0b8b82b82ab9"
server_key_version: "encryptionkey@0"
I20260929 20:46:57.467140 3538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a3e568cfaaa84e02ae618207bf2e3fe9"
format_stamp: "Formatted at 2026-09-29 20:46:57 on dist-test-slave-blv7"
server_key: "adba186d82cb77e76db51a8289a04a34"
server_key_iv: "7efa7d3b00f840619c2d0b8b82b82ab9"
server_key_version: "encryptionkey@0"
I20260929 20:46:57.472427 3538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 20:46:57.475728 3553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:57.476963 3538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260929 20:46:57.477108 3538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "a3e568cfaaa84e02ae618207bf2e3fe9"
format_stamp: "Formatted at 2026-09-29 20:46:57 on dist-test-slave-blv7"
server_key: "adba186d82cb77e76db51a8289a04a34"
server_key_iv: "7efa7d3b00f840619c2d0b8b82b82ab9"
server_key_version: "encryptionkey@0"
I20260929 20:46:57.477278 3538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:57.501850 3538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:57.505795 3538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:57.506080 3538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:57.507193 3538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:46:57.508452 3538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:46:57.508535 3538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:57.508653 3538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:46:57.508709 3538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:57.537307 3538 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43551
I20260929 20:46:57.537349 3665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43551 every 8 connection(s)
I20260929 20:46:57.538564 3538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:46:57.541620 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 3538
I20260929 20:46:57.541827 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:46:57.542635 388 external_mini_cluster.cc:1428] Setting key 87903247a8e15dcd479f30a8a38a601e
Sep 29 20:46:57 dist-test-slave-blv7 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:57.558977 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:57.540902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51315 (local address 127.0.97.62:39059)
0929 20:46:57.541372 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:57.541379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:57.542460 (+ 1081us) server_negotiation.cc:407] Connection header received
0929 20:46:57.544490 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:57.544496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:57.544575 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:57.544683 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:57.546566 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:57.547344 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:57.549373 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:57.549886 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:57.550014 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:57.553120 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:57.553126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:57.553130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:57.553168 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:57.556125 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:57.556780 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:57.556785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:57.556789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:57.556926 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:57.557281 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:57.557286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:57.557290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:57.557636 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:57.557784 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:57.558374 (+ 590us) server_negotiation.cc:299] Negotiation successful
0929 20:46:57.558609 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":229,"threads_started":1}
I20260929 20:46:57.559636 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:57.541471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39059 (local address 127.0.97.1:51315)
0929 20:46:57.542283 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:57.542337 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:57.544222 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:57.544945 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:57.544957 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:57.545628 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:57.546379 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:57.546403 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:57.547508 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:57.547513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:57.549221 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:57.549235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:57.550081 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:57.550085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:57.550265 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:57.550886 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:57.550906 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:57.552908 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:57.556254 (+ 3346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:57.556263 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:57.556279 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:57.556675 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:57.557022 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:57.557024 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:57.557026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:57.557187 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:57.557745 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:57.557750 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:57.558020 (+ 270us) client_negotiation.cc:769] Sending connection context
0929 20:46:57.558248 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 20:46:57.558595 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":299,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:46:57.560963 3666 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39059
I20260929 20:46:57.561255 3666 heartbeater.cc:464] Registering TS with master...
I20260929 20:46:57.561968 3666 heartbeater.cc:511] Master 127.0.97.62:39059 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 20:46:57.563944 3477 ts_manager.cc:194] Registered new tserver with Master: a3e568cfaaa84e02ae618207bf2e3fe9 (127.0.97.1:43551)
I20260929 20:46:57.565265 3477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:46:57.582486 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39059
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:57.878783 3672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:57.879067 3672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:57.879204 3672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:57.886153 3672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:57.886241 3672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:57.886384 3672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:46:57.895447 3672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:57.896545 3672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:57.897770 3672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:57.909684 3680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:57.909775 3678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:57.909740 3677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:57.910709 3672 server_base.cc:1034] running on GCE node
I20260929 20:46:57.911295 3672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:57.912166 3672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:57.913353 3672 hybrid_clock.cc:648] HybridClock initialized: now 1790714817913309 us; error 44 us; skew 500 ppm
I20260929 20:46:57.913635 3672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:57 dist-test-slave-blv7 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:57.916638 3672 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:46:57.917951 3672 webserver.cc:460] Webserver started at http://127.0.97.2:38699/ using document root <none> and password file <none>
I20260929 20:46:57.918570 3672 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:57.918674 3672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:57.918936 3672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:57.921811 3672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "08308f48a1c84aaca74dc5c31e12d75e"
format_stamp: "Formatted at 2026-09-29 20:46:57 on dist-test-slave-blv7"
server_key: "adbc05b6f6e00b2236334a936aa5c667"
server_key_iv: "5f723ba3807b96fc95e94409107b3192"
server_key_version: "encryptionkey@0"
I20260929 20:46:57.922531 3672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "08308f48a1c84aaca74dc5c31e12d75e"
format_stamp: "Formatted at 2026-09-29 20:46:57 on dist-test-slave-blv7"
server_key: "adbc05b6f6e00b2236334a936aa5c667"
server_key_iv: "5f723ba3807b96fc95e94409107b3192"
server_key_version: "encryptionkey@0"
I20260929 20:46:57.927277 3672 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260929 20:46:57.930404 3687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:57.931684 3672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:46:57.931818 3672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "08308f48a1c84aaca74dc5c31e12d75e"
format_stamp: "Formatted at 2026-09-29 20:46:57 on dist-test-slave-blv7"
server_key: "adbc05b6f6e00b2236334a936aa5c667"
server_key_iv: "5f723ba3807b96fc95e94409107b3192"
server_key_version: "encryptionkey@0"
I20260929 20:46:57.931998 3672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:57.956736 3672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:57.960685 3672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:57.960944 3672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:57.961781 3672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:46:57.963105 3672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:46:57.963181 3672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:57.963291 3672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:46:57.963340 3672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:57.992187 3672 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34785
I20260929 20:46:57.992236 3799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34785 every 8 connection(s)
I20260929 20:46:57.993348 3672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:46:57.996307 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 3672
I20260929 20:46:57.996505 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:46:57.997023 388 external_mini_cluster.cc:1428] Setting key 87962f9cdcca21081c1960b9408fec4d
Sep 29 20:46:58 dist-test-slave-blv7 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:58.014170 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:57.995591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32951 (local address 127.0.97.62:39059)
0929 20:46:57.995746 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:57.995750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:57.996979 (+ 1229us) server_negotiation.cc:407] Connection header received
0929 20:46:57.999128 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:57.999132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:57.999193 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:57.999270 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:58.001157 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:58.001872 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:58.003919 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:58.004330 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:58.004435 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:58.008134 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:58.008141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:58.008145 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:58.008181 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:58.011212 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:58.011875 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:58.011881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:58.011885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:58.012035 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:58.012429 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:58.012433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:58.012436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:58.012757 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:58.012898 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:58.013591 (+ 693us) server_negotiation.cc:299] Negotiation successful
0929 20:46:58.013815 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:46:58.015152 3802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:57.996017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39059 (local address 127.0.97.2:32951)
0929 20:46:57.996808 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:57.996856 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:57.998818 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:57.999517 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:57.999529 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:58.000212 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:58.001010 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:58.001034 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:58.002022 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:58.002026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:58.003790 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:58.003804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:58.004532 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:58.004539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:58.004756 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:58.005407 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:58.005433 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:58.007889 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:58.011332 (+ 3443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:58.011339 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:58.011351 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:58.011744 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:58.012151 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:58.012155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:58.012157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:58.012324 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:58.012901 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:58.012909 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:58.013228 (+ 319us) client_negotiation.cc:769] Sending connection context
0929 20:46:58.013461 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 20:46:58.013946 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":289,"threads_started":1}
I20260929 20:46:58.016563 3800 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39059
I20260929 20:46:58.016850 3800 heartbeater.cc:464] Registering TS with master...
I20260929 20:46:58.017602 3800 heartbeater.cc:511] Master 127.0.97.62:39059 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 20:46:58.018980 3477 ts_manager.cc:194] Registered new tserver with Master: 08308f48a1c84aaca74dc5c31e12d75e (127.0.97.2:34785)
I20260929 20:46:58.019760 3477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:32951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:46:58.038431 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39059
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:46:58.332732 3805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:46:58.332989 3805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:46:58.333102 3805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:46:58.340256 3805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:46:58.340359 3805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:46:58.340499 3805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:46:58.349767 3805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:46:58.350987 3805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:46:58.352195 3805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:46:58.364018 3811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:58.364101 3810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:46:58.364079 3813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:46:58.364913 3805 server_base.cc:1034] running on GCE node
I20260929 20:46:58.365494 3805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:46:58.366320 3805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:46:58.367509 3805 hybrid_clock.cc:648] HybridClock initialized: now 1790714818367473 us; error 27 us; skew 500 ppm
I20260929 20:46:58.367767 3805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:46:58 dist-test-slave-blv7 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:46:58.370546 3805 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:46:58.371866 3805 webserver.cc:460] Webserver started at http://127.0.97.3:38077/ using document root <none> and password file <none>
I20260929 20:46:58.372466 3805 fs_manager.cc:359] Metadata directory not provided
I20260929 20:46:58.372553 3805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:46:58.372774 3805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:46:58.375523 3805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "dbb96ca6d6ac41a2b3421161208a3383"
format_stamp: "Formatted at 2026-09-29 20:46:58 on dist-test-slave-blv7"
server_key: "cd9068ff594701894cf1b9fc02101646"
server_key_iv: "ba29fd90578b50f94adadd2f53dc1b26"
server_key_version: "encryptionkey@0"
I20260929 20:46:58.376216 3805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "dbb96ca6d6ac41a2b3421161208a3383"
format_stamp: "Formatted at 2026-09-29 20:46:58 on dist-test-slave-blv7"
server_key: "cd9068ff594701894cf1b9fc02101646"
server_key_iv: "ba29fd90578b50f94adadd2f53dc1b26"
server_key_version: "encryptionkey@0"
I20260929 20:46:58.380873 3805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 20:46:58.384191 3820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:58.385529 3805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:46:58.385669 3805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "dbb96ca6d6ac41a2b3421161208a3383"
format_stamp: "Formatted at 2026-09-29 20:46:58 on dist-test-slave-blv7"
server_key: "cd9068ff594701894cf1b9fc02101646"
server_key_iv: "ba29fd90578b50f94adadd2f53dc1b26"
server_key_version: "encryptionkey@0"
I20260929 20:46:58.385843 3805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:46:58.398386 3805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:46:58.402093 3805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:46:58.402340 3805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:46:58.403218 3805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:46:58.404495 3805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:46:58.404561 3805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:58.404670 3805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:46:58.404717 3805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:46:58.433612 3805 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37145
I20260929 20:46:58.433656 3932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37145 every 8 connection(s)
I20260929 20:46:58.434834 3805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:46:58.442903 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 3805
I20260929 20:46:58.443089 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:46:58.443630 388 external_mini_cluster.cc:1428] Setting key e7ba42d5736d2ba366db93d6283a3c6c
Sep 29 20:46:58 dist-test-slave-blv7 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:58.455407 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:58.437284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46867 (local address 127.0.97.62:39059)
0929 20:46:58.437462 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:58.437466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:58.438570 (+ 1104us) server_negotiation.cc:407] Connection header received
0929 20:46:58.440266 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:58.440270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:58.440334 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:58.440411 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:58.442124 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:58.443300 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:58.445800 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:58.446409 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:58.446525 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:58.450246 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:58.450252 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:58.450256 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:58.450289 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:58.452666 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:58.453316 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:58.453322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:58.453326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:58.453433 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:58.453811 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:58.453816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:58.453819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:58.454074 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:58.454194 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:58.454843 (+ 649us) server_negotiation.cc:299] Negotiation successful
0929 20:46:58.455081 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:46:58.456346 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:58.437727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39059 (local address 127.0.97.3:46867)
0929 20:46:58.438387 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0929 20:46:58.438422 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:46:58.440019 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:46:58.440655 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:46:58.440663 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:46:58.441228 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:46:58.441944 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:58.441965 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:58.443497 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:58.443503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:58.445644 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:46:58.445659 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:46:58.446857 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:46:58.446864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:46:58.447309 (+ 445us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:58.447977 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:46:58.448000 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:46:58.450050 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:46:58.452788 (+ 2738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:58.452796 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:58.452814 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:58.453212 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:58.453526 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:46:58.453531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:46:58.453535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:46:58.453711 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:46:58.454188 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:46:58.454196 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:46:58.454505 (+ 309us) client_negotiation.cc:769] Sending connection context
0929 20:46:58.454741 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 20:46:58.455282 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":255,"threads_started":1}
I20260929 20:46:58.457723 3933 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39059
I20260929 20:46:58.458132 3933 heartbeater.cc:464] Registering TS with master...
I20260929 20:46:58.458881 3933 heartbeater.cc:511] Master 127.0.97.62:39059 requested a full tablet report, sending...
I20260929 20:46:58.460326 3477 ts_manager.cc:194] Registered new tserver with Master: dbb96ca6d6ac41a2b3421161208a3383 (127.0.97.3:37145)
I20260929 20:46:58.461122 3477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46867
I20260929 20:46:58.472101 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:46:58 dist-test-slave-blv7 krb5kdc[3432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714818, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 20:46:58 dist-test-slave-blv7 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714818, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:46:58.513792 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:58.500807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.0.97.62:39059)
0929 20:46:58.500976 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:58.500981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:58.501372 (+ 391us) server_negotiation.cc:407] Connection header received
0929 20:46:58.501520 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:58.501524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:58.501606 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:58.501690 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:58.502983 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:58.503709 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:58.505032 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:58.505427 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:58.505622 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:58.509262 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:58.509267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:58.509270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:58.509301 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:58.511755 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:58.512310 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:58.512313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:58.512316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:58.512424 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:58.512725 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:58.512728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:58.512730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:58.513033 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:58.513207 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:58.513420 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 20:46:58.513578 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 20:46:58.514263 3477 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55650
I20260929 20:46:58.515450 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 3538
Sep 29 20:46:58 dist-test-slave-blv7 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714818, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:46:58.530303 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:58.516479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58600 (local address 127.0.97.1:43551)
0929 20:46:58.516942 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:58.516951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:58.516980 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:46:58.517039 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:58.517045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:58.517249 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:58.517366 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:58.518474 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:58.519656 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:58.520891 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:58.521569 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:58.521708 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:58.525365 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:58.525377 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:58.525391 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:58.525426 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:58.528046 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:58.528647 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:58.528654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:58.528656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:58.528771 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:58.529047 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:58.529051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:58.529054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:58.529352 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:58.529572 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:58.529776 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 20:46:58.529958 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":220,"threads_started":1}
W20260929 20:46:58.531544 3661 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58600
W20260929 20:46:58.532567 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 3538: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:46:58.532737 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 3538
I20260929 20:46:58.544685 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 3672
Sep 29 20:46:58 dist-test-slave-blv7 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714818, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:46:58.559244 3945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:58.545578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40680 (local address 127.0.97.2:34785)
0929 20:46:58.546137 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:58.546144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:58.546171 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:46:58.546233 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:58.546240 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:58.546453 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:58.546585 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:58.547690 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:58.548773 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:58.549909 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:58.550450 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:58.550559 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:58.554099 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:58.554110 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:58.554125 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:58.554157 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:58.556792 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:58.557348 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:58.557354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:58.557357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:58.557458 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:58.557740 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:58.557743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:58.557745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:58.558030 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:58.558248 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:58.558606 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 20:46:58.558852 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":288,"threads_started":1}
W20260929 20:46:58.560416 3795 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40680
W20260929 20:46:58.561446 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 3672: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:46:58.561569 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 3672
I20260929 20:46:58.573137 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 3805
Sep 29 20:46:58 dist-test-slave-blv7 krb5kdc[3432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714818, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:46:58.587469 3946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:46:58.573871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58376 (local address 127.0.97.3:37145)
0929 20:46:58.574438 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0929 20:46:58.574444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:46:58.574469 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:46:58.574519 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:46:58.574525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:46:58.574716 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:46:58.574847 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:46:58.576053 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:58.577021 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:58.578182 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:46:58.578704 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:46:58.578811 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:46:58.582539 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:46:58.582549 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:46:58.582561 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:46:58.582589 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:46:58.585031 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:58.585601 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:58.585607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:58.585609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:58.585715 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:46:58.586023 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:46:58.586028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:46:58.586031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:46:58.586329 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:46:58.586599 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 20:46:58.586888 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 20:46:58.587157 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":279,"threads_started":1}
W20260929 20:46:58.588624 3928 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58376
W20260929 20:46:58.589653 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 3805: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:46:58.589789 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 3805
I20260929 20:46:58.601766 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 3445
I20260929 20:46:58.602756 3507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:46:58.699678 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 3445
2026-09-29T20:46:58Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3278 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:46:58 dist-test-slave-blv7 krb5kdc[3950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:46:58 dist-test-slave-blv7 krb5kdc[3950](info): set up 2 sockets
Sep 29 20:46:58 dist-test-slave-blv7 krb5kdc[3950](info): commencing operation
krb5kdc: starting...
W20260929 20:46:59.687067 388 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:46:59 dist-test-slave-blv7 krb5kdc[3950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:46:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:46:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:46:59.858449 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41035
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:00.151993 3963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:00.152256 3963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:00.152377 3963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:00.159386 3963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:47:00.159502 3963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:00.159572 3963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:00.159649 3963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:47:00.159703 3963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:47:00.169113 3963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:00.170325 3963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:00.171592 3963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:00.182266 3971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:00.182368 3968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:00.182344 3969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:00.183396 3963 server_base.cc:1034] running on GCE node
I20260929 20:47:00.184053 3963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:00.185266 3963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:00.186446 3963 hybrid_clock.cc:648] HybridClock initialized: now 1790714820186383 us; error 55 us; skew 500 ppm
I20260929 20:47:00.186759 3963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:00 dist-test-slave-blv7 krb5kdc[3950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:00.189603 3963 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:47:00.190896 3963 webserver.cc:460] Webserver started at http://127.0.97.62:36101/ using document root <none> and password file <none>
I20260929 20:47:00.191587 3963 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:00.191681 3963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:00.191944 3963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:00.194612 3963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "5435fab4935b4b15bf23b4bc004842a9"
format_stamp: "Formatted at 2026-09-29 20:47:00 on dist-test-slave-blv7"
server_key: "3b1fc0164a66aed203a9798984f867e2"
server_key_iv: "8ac184a1f3a2413c2a1056715933a9eb"
server_key_version: "encryptionkey@0"
I20260929 20:47:00.195387 3963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "5435fab4935b4b15bf23b4bc004842a9"
format_stamp: "Formatted at 2026-09-29 20:47:00 on dist-test-slave-blv7"
server_key: "3b1fc0164a66aed203a9798984f867e2"
server_key_iv: "8ac184a1f3a2413c2a1056715933a9eb"
server_key_version: "encryptionkey@0"
I20260929 20:47:00.200203 3963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:47:00.203368 3978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:00.204588 3963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:47:00.204736 3963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "5435fab4935b4b15bf23b4bc004842a9"
format_stamp: "Formatted at 2026-09-29 20:47:00 on dist-test-slave-blv7"
server_key: "3b1fc0164a66aed203a9798984f867e2"
server_key_iv: "8ac184a1f3a2413c2a1056715933a9eb"
server_key_version: "encryptionkey@0"
I20260929 20:47:00.204901 3963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:00.226831 3963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:00.230564 3963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:00.230798 3963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:00.246646 3963 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41035
I20260929 20:47:00.246672 4029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41035 every 8 connection(s)
I20260929 20:47:00.247857 3963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:47:00.251353 4030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:00.252374 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 3963
I20260929 20:47:00.252573 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:47:00.253180 388 external_mini_cluster.cc:1428] Setting key 1135ea3c604c84f8298353a3aed24dc8
I20260929 20:47:00.264335 4030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9: Bootstrap starting.
Sep 29 20:47:00 dist-test-slave-blv7 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:47:00.267833 4030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:00.269063 4030 log.cc:824] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:00.272239 4030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9: No bootstrap required, opened a new log
I20260929 20:47:00.275590 4033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:00.254987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47680 (local address 127.0.97.62:41035)
0929 20:47:00.255751 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:00.255763 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:00.255803 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:47:00.257313 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:00.257334 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:00.257737 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:00.258052 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:00.259852 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:00.261260 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:00.262975 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:00.263679 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:00.263806 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:00.268209 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:00.268232 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:00.268257 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:00.268308 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:00.271862 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:00.272465 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:00.272471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:00.272477 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:00.272638 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:00.272994 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:00.272999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:00.273001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:00.273488 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:00.273774 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:00.274213 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 20:47:00.274588 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":343,"threads_started":1}
I20260929 20:47:00.279667 4030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5435fab4935b4b15bf23b4bc004842a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41035 } }
I20260929 20:47:00.279960 4030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:00.280049 4030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5435fab4935b4b15bf23b4bc004842a9, State: Initialized, Role: FOLLOWER
I20260929 20:47:00.280571 4030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5435fab4935b4b15bf23b4bc004842a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41035 } }
I20260929 20:47:00.280781 4030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:47:00.280892 4030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:47:00.281028 4030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:00.283288 4030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5435fab4935b4b15bf23b4bc004842a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41035 } }
I20260929 20:47:00.283725 4030 leader_election.cc:302] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5435fab4935b4b15bf23b4bc004842a9; no voters:
I20260929 20:47:00.284232 4030 leader_election.cc:288] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:47:00.284353 4035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:47:00.285238 4035 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9 [term 1 LEADER]: Becoming Leader. State: Replica: 5435fab4935b4b15bf23b4bc004842a9, State: Running, Role: LEADER
I20260929 20:47:00.285689 4035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5435fab4935b4b15bf23b4bc004842a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41035 } }
I20260929 20:47:00.286084 4030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:47:00.288298 4037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5435fab4935b4b15bf23b4bc004842a9. Latest consensus state: current_term: 1 leader_uuid: "5435fab4935b4b15bf23b4bc004842a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5435fab4935b4b15bf23b4bc004842a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41035 } } }
I20260929 20:47:00.288288 4036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5435fab4935b4b15bf23b4bc004842a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5435fab4935b4b15bf23b4bc004842a9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41035 } } }
I20260929 20:47:00.288771 4037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9 [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:00.288976 4036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9 [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:00.289754 4044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:47:00.298081 4044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:47:00.306133 4044 catalog_manager.cc:1347] Generated new cluster ID: db3c4fdb99b54aeebb4d77414df0899c
I20260929 20:47:00.306237 4044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:47:00.319394 4044 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:47:00.321130 4044 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:47:00.329061 4044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5435fab4935b4b15bf23b4bc004842a9: Generated new TSK 0
I20260929 20:47:00.329963 4044 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:47:00.370603 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41035
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:00.677007 4056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:00.677263 4056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:00.677384 4056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:00.684528 4056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:00.684623 4056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:00.684760 4056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:47:00.694597 4056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:00.695863 4056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:00.697163 4056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:00.709211 4061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:00.709286 4062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:00.709270 4064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:00.710048 4056 server_base.cc:1034] running on GCE node
I20260929 20:47:00.710587 4056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:00.711486 4056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:00.712677 4056 hybrid_clock.cc:648] HybridClock initialized: now 1790714820712635 us; error 38 us; skew 500 ppm
I20260929 20:47:00.712965 4056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:00 dist-test-slave-blv7 krb5kdc[3950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:00.721493 4056 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:47:00.722774 4056 webserver.cc:460] Webserver started at http://127.0.97.1:45221/ using document root <none> and password file <none>
I20260929 20:47:00.723464 4056 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:00.723562 4056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:00.723798 4056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:00.726540 4056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "8a650e64f74f482fa6507af603b29c85"
format_stamp: "Formatted at 2026-09-29 20:47:00 on dist-test-slave-blv7"
server_key: "bb60d9fb34bece9b264e75d8083bbcaf"
server_key_iv: "85f43fd2acba4b325e864fa98107ce6d"
server_key_version: "encryptionkey@0"
I20260929 20:47:00.727273 4056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8a650e64f74f482fa6507af603b29c85"
format_stamp: "Formatted at 2026-09-29 20:47:00 on dist-test-slave-blv7"
server_key: "bb60d9fb34bece9b264e75d8083bbcaf"
server_key_iv: "85f43fd2acba4b325e864fa98107ce6d"
server_key_version: "encryptionkey@0"
I20260929 20:47:00.731861 4056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:47:00.734980 4071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:00.736222 4056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:47:00.736342 4056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "8a650e64f74f482fa6507af603b29c85"
format_stamp: "Formatted at 2026-09-29 20:47:00 on dist-test-slave-blv7"
server_key: "bb60d9fb34bece9b264e75d8083bbcaf"
server_key_iv: "85f43fd2acba4b325e864fa98107ce6d"
server_key_version: "encryptionkey@0"
I20260929 20:47:00.736488 4056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:00.753814 4056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:00.757742 4056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:00.757956 4056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:00.758759 4056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:00.759934 4056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:00.760005 4056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:00.760094 4056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:00.760136 4056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:00.790741 4056 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37639
I20260929 20:47:00.790783 4183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37639 every 8 connection(s)
I20260929 20:47:00.791952 4056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:47:00.795286 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 4056
I20260929 20:47:00.795466 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:47:00.796090 388 external_mini_cluster.cc:1428] Setting key 914af3d11e94e4b10c645ff222119685
Sep 29 20:47:00 dist-test-slave-blv7 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:47:00.811386 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:00.794241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60927 (local address 127.0.97.62:41035)
0929 20:47:00.794640 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:00.794644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:00.795677 (+ 1033us) server_negotiation.cc:407] Connection header received
0929 20:47:00.797576 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:00.797582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:00.797644 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:00.797749 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:00.799222 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:00.799912 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:00.801607 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:00.802167 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:00.802289 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:00.806085 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:00.806090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:00.806093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:00.806151 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:00.808694 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:00.809336 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:00.809341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:00.809343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:00.809450 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:00.809828 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:00.809832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:00.809835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:00.810083 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:00.810199 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:00.810820 (+ 621us) server_negotiation.cc:299] Negotiation successful
0929 20:47:00.811044 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":194,"threads_started":1}
I20260929 20:47:00.812266 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:00.794725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41035 (local address 127.0.97.1:60927)
0929 20:47:00.795513 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:00.795563 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:00.797337 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:00.797996 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:00.798004 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:00.798500 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:00.799066 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:00.799090 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:00.800062 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:00.800066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:00.801474 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:00.801488 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:00.802377 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:00.802384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:00.802644 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:00.803316 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:00.803345 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:00.805858 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:00.808811 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:00.808820 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:00.808836 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:00.809221 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:00.809543 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:00.809548 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:00.809553 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:00.809725 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:00.810193 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:00.810200 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:00.810502 (+ 302us) client_negotiation.cc:769] Sending connection context
0929 20:47:00.810721 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 20:47:00.811227 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":349,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:47:00.813649 4184 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41035
I20260929 20:47:00.813912 4184 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:00.814692 4184 heartbeater.cc:511] Master 127.0.97.62:41035 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 20:47:00.816668 3995 ts_manager.cc:194] Registered new tserver with Master: 8a650e64f74f482fa6507af603b29c85 (127.0.97.1:37639)
I20260929 20:47:00.818001 3995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:47:00.834266 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41035
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:01.126688 4190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:01.126940 4190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:01.127084 4190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:01.133970 4190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:01.134064 4190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:01.134207 4190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:47:01.143424 4190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:01.144586 4190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:01.145772 4190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:01.157518 4196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:01.157608 4195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:01.157716 4198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:01.158245 4190 server_base.cc:1034] running on GCE node
I20260929 20:47:01.158772 4190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:01.159687 4190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:01.160888 4190 hybrid_clock.cc:648] HybridClock initialized: now 1790714821160840 us; error 43 us; skew 500 ppm
I20260929 20:47:01.161172 4190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:01 dist-test-slave-blv7 krb5kdc[3950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:01.163993 4190 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:47:01.165249 4190 webserver.cc:460] Webserver started at http://127.0.97.2:33205/ using document root <none> and password file <none>
I20260929 20:47:01.165858 4190 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:01.165954 4190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:01.166219 4190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:01.168911 4190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "c136cc59d3164009a37364d42c2105a7"
format_stamp: "Formatted at 2026-09-29 20:47:01 on dist-test-slave-blv7"
server_key: "42ff7868ddb119d223e0600eae7ff1a7"
server_key_iv: "8366dda16e1bc8df82ff855a6a89df6f"
server_key_version: "encryptionkey@0"
I20260929 20:47:01.169610 4190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c136cc59d3164009a37364d42c2105a7"
format_stamp: "Formatted at 2026-09-29 20:47:01 on dist-test-slave-blv7"
server_key: "42ff7868ddb119d223e0600eae7ff1a7"
server_key_iv: "8366dda16e1bc8df82ff855a6a89df6f"
server_key_version: "encryptionkey@0"
I20260929 20:47:01.174266 4190 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 20:47:01.177616 4205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:01.178802 4190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 20:47:01.178979 4190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "c136cc59d3164009a37364d42c2105a7"
format_stamp: "Formatted at 2026-09-29 20:47:01 on dist-test-slave-blv7"
server_key: "42ff7868ddb119d223e0600eae7ff1a7"
server_key_iv: "8366dda16e1bc8df82ff855a6a89df6f"
server_key_version: "encryptionkey@0"
I20260929 20:47:01.179201 4190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:01.196540 4190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:01.200088 4190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:01.200313 4190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:01.201138 4190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:01.202358 4190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:01.202427 4190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:01.202538 4190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:01.202586 4190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:01.230090 4190 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42851
I20260929 20:47:01.230131 4317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42851 every 8 connection(s)
I20260929 20:47:01.231369 4190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:47:01.238271 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 4190
I20260929 20:47:01.238451 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:47:01.238976 388 external_mini_cluster.cc:1428] Setting key 68d55242f79b33f809ca4a248455db8d
Sep 29 20:47:01 dist-test-slave-blv7 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:47:01.251634 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:01.233635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47737 (local address 127.0.97.62:41035)
0929 20:47:01.233840 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:01.233846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:01.235095 (+ 1249us) server_negotiation.cc:407] Connection header received
0929 20:47:01.236880 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:01.236884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:01.236946 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:01.237029 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:01.238382 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:01.239417 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:01.241203 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:01.241839 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:01.242005 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:01.245211 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:01.245218 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:01.245223 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:01.245265 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:01.248535 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:01.249295 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:01.249301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:01.249304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:01.249460 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:01.249862 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:01.249867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:01.249871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:01.250199 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:01.250326 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:01.251099 (+ 773us) server_negotiation.cc:299] Negotiation successful
0929 20:47:01.251316 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:47:01.252672 4320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:01.234149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41035 (local address 127.0.97.2:47737)
0929 20:47:01.234918 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:01.234969 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:01.236648 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:01.237214 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:01.237221 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:01.237710 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:01.238226 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:01.238242 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:01.239581 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:01.239586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:01.241042 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:01.241053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:01.242093 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:01.242098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:01.242282 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:01.242953 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:01.242981 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:01.244990 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:01.248665 (+ 3675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:01.248673 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:01.248688 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:01.249189 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:01.249558 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:01.249562 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:01.249565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:01.249761 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:01.250309 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:01.250316 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:01.250696 (+ 380us) client_negotiation.cc:769] Sending connection context
0929 20:47:01.250970 (+ 274us) client_negotiation.cc:240] Negotiation successful
0929 20:47:01.251453 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":339,"threads_started":1}
I20260929 20:47:01.253973 4318 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41035
I20260929 20:47:01.254240 4318 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:01.254895 4318 heartbeater.cc:511] Master 127.0.97.62:41035 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:47:01.256166 3995 ts_manager.cc:194] Registered new tserver with Master: c136cc59d3164009a37364d42c2105a7 (127.0.97.2:42851)
I20260929 20:47:01.256964 3995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:47:01.278630 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41035
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:01.574456 4323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:01.574735 4323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:01.574872 4323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:01.581858 4323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:01.581952 4323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:01.582084 4323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:47:01.591606 4323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41035
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:01.592720 4323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:01.593972 4323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:01.605501 4328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:01.606128 4329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:01.606326 4331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:01.606741 4323 server_base.cc:1034] running on GCE node
I20260929 20:47:01.607306 4323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:01.608129 4323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:01.609344 4323 hybrid_clock.cc:648] HybridClock initialized: now 1790714821609277 us; error 34 us; skew 500 ppm
I20260929 20:47:01.609675 4323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:01 dist-test-slave-blv7 krb5kdc[3950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:01.612466 4323 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:47:01.613693 4323 webserver.cc:460] Webserver started at http://127.0.97.3:36377/ using document root <none> and password file <none>
I20260929 20:47:01.614320 4323 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:01.614456 4323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:01.614739 4323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:01.617376 4323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "91e0ca6d28dc49f79547afafece14ddf"
format_stamp: "Formatted at 2026-09-29 20:47:01 on dist-test-slave-blv7"
server_key: "0756f63eaf6f5a4a99c514499c80a155"
server_key_iv: "afd463f4dc2cfd5f8b7684f52267cac6"
server_key_version: "encryptionkey@0"
I20260929 20:47:01.618077 4323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "91e0ca6d28dc49f79547afafece14ddf"
format_stamp: "Formatted at 2026-09-29 20:47:01 on dist-test-slave-blv7"
server_key: "0756f63eaf6f5a4a99c514499c80a155"
server_key_iv: "afd463f4dc2cfd5f8b7684f52267cac6"
server_key_version: "encryptionkey@0"
I20260929 20:47:01.622743 4323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.007s
I20260929 20:47:01.625890 4338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:01.627286 4323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 20:47:01.627437 4323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "91e0ca6d28dc49f79547afafece14ddf"
format_stamp: "Formatted at 2026-09-29 20:47:01 on dist-test-slave-blv7"
server_key: "0756f63eaf6f5a4a99c514499c80a155"
server_key_iv: "afd463f4dc2cfd5f8b7684f52267cac6"
server_key_version: "encryptionkey@0"
I20260929 20:47:01.627624 4323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:01.662428 4323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:01.666160 4323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:01.666379 4323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:01.667289 4323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:01.668526 4323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:01.668596 4323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:01.668676 4323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:01.668717 4323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:01.696923 4323 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37815
I20260929 20:47:01.696964 4450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37815 every 8 connection(s)
I20260929 20:47:01.698206 4323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:47:01.703356 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 4323
I20260929 20:47:01.703516 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:47:01.703948 388 external_mini_cluster.cc:1428] Setting key 2d7cdc1485457060b3ef3e63b6aa8b7f
Sep 29 20:47:01 dist-test-slave-blv7 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:47:01.716789 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:01.700659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57113 (local address 127.0.97.62:41035)
0929 20:47:01.700844 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:01.700850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:01.702011 (+ 1161us) server_negotiation.cc:407] Connection header received
0929 20:47:01.703804 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:01.703810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:01.703887 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:01.704001 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:01.705820 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:01.706632 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:01.708201 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:01.708606 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:01.708705 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:01.711832 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:01.711836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:01.711840 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:01.711870 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:01.714113 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:01.714787 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:01.714790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:01.714793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:01.714899 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:01.715304 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:01.715307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:01.715309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:01.715550 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:01.715662 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:01.716391 (+ 729us) server_negotiation.cc:299] Negotiation successful
0929 20:47:01.716537 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:47:01.717868 4453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:01.701195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41035 (local address 127.0.97.3:57113)
0929 20:47:01.701865 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:01.701907 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:01.703547 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:01.704207 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:01.704215 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:01.704803 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:01.705675 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:01.705696 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:01.706765 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:01.706770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:01.708096 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:01.708107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:01.708784 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:01.708790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:01.708994 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:01.709546 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:01.709570 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:01.711663 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:01.714239 (+ 2576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:01.714246 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:01.714258 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:01.714693 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:01.715003 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:01.715006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:01.715009 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:01.715219 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:01.715743 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:01.715749 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:01.716093 (+ 344us) client_negotiation.cc:769] Sending connection context
0929 20:47:01.716325 (+ 232us) client_negotiation.cc:240] Negotiation successful
0929 20:47:01.716673 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":278,"threads_started":1}
I20260929 20:47:01.719266 4451 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41035
I20260929 20:47:01.719583 4451 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:01.720217 4451 heartbeater.cc:511] Master 127.0.97.62:41035 requested a full tablet report, sending...
I20260929 20:47:01.721439 3995 ts_manager.cc:194] Registered new tserver with Master: 91e0ca6d28dc49f79547afafece14ddf (127.0.97.3:37815)
I20260929 20:47:01.722189 3995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57113
I20260929 20:47:01.731482 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:47:01 dist-test-slave-blv7 krb5kdc[3950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:47:01 dist-test-slave-blv7 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:47:01.774322 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:01.760267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33702 (local address 127.0.97.1:37639)
0929 20:47:01.760763 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:01.760772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:01.760803 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:47:01.760949 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:01.760955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:01.761178 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:01.761311 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:01.762611 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:01.763638 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:01.764798 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:01.765323 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:01.765430 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:01.768847 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:01.768857 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:01.768872 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:01.768902 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:01.771371 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:01.772072 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:01.772082 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:01.772085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:01.772232 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:01.772678 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:01.772683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:01.772686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:01.773110 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:01.773387 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:01.773680 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 20:47:01.773925 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":205,"threads_started":1}
Sep 29 20:47:01 dist-test-slave-blv7 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:47:01.790709 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:01.778246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47684 (local address 127.0.97.62:41035)
0929 20:47:01.778433 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:01.778438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:01.778611 (+ 173us) server_negotiation.cc:407] Connection header received
0929 20:47:01.778786 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:01.778791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:01.778856 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:01.778930 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:01.780100 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:01.780799 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:01.781984 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:01.782384 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:01.782483 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:01.785743 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:01.785747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:01.785750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:01.785780 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:01.788330 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:01.788856 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:01.788860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:01.788862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:01.788982 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:01.789245 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:01.789248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:01.789250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:01.789623 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:01.789826 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:01.790110 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 20:47:01.790346 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 20:47:01.791241 3995 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47684
I20260929 20:47:01.792086 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 4056
I20260929 20:47:01.804117 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:01.793030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.0.97.1:37639)
0929 20:47:01.793223 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:01.793228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:01.793536 (+ 308us) server_negotiation.cc:407] Connection header received
0929 20:47:01.793724 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:01.793727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:01.793801 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:01.793878 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:01.794999 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:01.795719 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:01.796862 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:01.797297 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:01.797402 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:01.799543 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:01.799548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:01.799551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:01.799577 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:01.801851 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:01.802426 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:01.802431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:01.802435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:01.802548 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:01.802857 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:01.802862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:01.802866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:01.803139 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:01.803335 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:01.803637 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 20:47:01.803832 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:47:01.804638 4179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:01.882639 4184 heartbeater.cc:503] Master 127.0.97.62:41035 was elected leader, sending a full tablet report...
I20260929 20:47:01.883122 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 4056
W20260929 20:47:01.895746 3982 connection.cc:535] server connection from 127.0.97.1:60927 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 20:47:01.895915 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 4190
Sep 29 20:47:01 dist-test-slave-blv7 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:47:01.910641 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:01.896818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58362 (local address 127.0.97.2:42851)
0929 20:47:01.897376 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:01.897383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:01.897407 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:47:01.897464 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:01.897470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:01.897685 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:01.897825 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:01.899061 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:01.900038 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:01.901215 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:01.901734 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:01.901841 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:01.905507 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:01.905517 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:01.905531 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:01.905569 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:01.908143 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:01.908710 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:01.908717 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:01.908719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:01.908830 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:01.909186 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:01.909191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:01.909194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:01.909500 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:01.909774 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:01.910062 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 20:47:01.910301 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":317,"threads_started":1}
I20260929 20:47:01.911841 4313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:01.989841 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 4190
I20260929 20:47:02.001986 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 4323
Sep 29 20:47:02 dist-test-slave-blv7 krb5kdc[3950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:47:02.016991 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:02.002861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44076 (local address 127.0.97.3:37815)
0929 20:47:02.003509 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:02.003516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:02.003549 (+ 33us) server_negotiation.cc:407] Connection header received
0929 20:47:02.003609 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:02.003615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:02.003814 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:02.003940 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:02.005151 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:02.006165 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:02.007612 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:02.008198 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:02.008293 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:02.012214 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:02.012224 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:02.012239 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:02.012274 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:02.014705 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:02.015389 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:02.015396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:02.015399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:02.015510 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:02.015773 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:02.015776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:02.015778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:02.016077 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:02.016302 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:02.016528 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 20:47:02.016724 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":371,"threads_started":1}
I20260929 20:47:02.018194 4446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:02.097328 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 4323
I20260929 20:47:02.110138 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 3963
I20260929 20:47:02.111397 4025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:02.209324 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 3963
2026-09-29T20:47:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3510 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:47:02 dist-test-slave-blv7 krb5kdc[4470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:47:02 dist-test-slave-blv7 krb5kdc[4470](info): set up 2 sockets
Sep 29 20:47:02 dist-test-slave-blv7 krb5kdc[4470](info): commencing operation
krb5kdc: starting...
W20260929 20:47:03.207073 388 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:47:03 dist-test-slave-blv7 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:47:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:47:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:47:03.378242 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41411
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:03.672955 4483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:03.673241 4483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:03.673372 4483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:03.680351 4483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:47:03.680460 4483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:03.680536 4483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:03.680608 4483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:47:03.680687 4483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:47:03.690140 4483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:03.691426 4483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:03.692627 4483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:03.703269 4488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:03.703241 4491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:03.703297 4489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:03.704396 4483 server_base.cc:1034] running on GCE node
I20260929 20:47:03.705097 4483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:03.706390 4483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:03.707568 4483 hybrid_clock.cc:648] HybridClock initialized: now 1790714823707521 us; error 43 us; skew 500 ppm
I20260929 20:47:03.707868 4483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:03 dist-test-slave-blv7 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:03.710726 4483 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:47:03.711896 4483 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:03.712002 4483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:03.712282 4483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:03.715065 4483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "3cdce833cc084f50bc62e60e2cf11b69"
format_stamp: "Formatted at 2026-09-29 20:47:03 on dist-test-slave-blv7"
server_key: "0d944136195576fe114011cdb12ff701"
server_key_iv: "740c15bed25db9694e160e22a9b0e2b7"
server_key_version: "encryptionkey@0"
I20260929 20:47:03.715786 4483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "3cdce833cc084f50bc62e60e2cf11b69"
format_stamp: "Formatted at 2026-09-29 20:47:03 on dist-test-slave-blv7"
server_key: "0d944136195576fe114011cdb12ff701"
server_key_iv: "740c15bed25db9694e160e22a9b0e2b7"
server_key_version: "encryptionkey@0"
I20260929 20:47:03.720469 4483 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:47:03.723659 4497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:03.724871 4483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:47:03.725059 4483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "3cdce833cc084f50bc62e60e2cf11b69"
format_stamp: "Formatted at 2026-09-29 20:47:03 on dist-test-slave-blv7"
server_key: "0d944136195576fe114011cdb12ff701"
server_key_iv: "740c15bed25db9694e160e22a9b0e2b7"
server_key_version: "encryptionkey@0"
I20260929 20:47:03.725242 4483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:03.743099 4483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:03.746989 4483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:03.747258 4483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:03.763185 4483 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41411
I20260929 20:47:03.763168 4548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41411 every 8 connection(s)
I20260929 20:47:03.764027 4483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:47:03.767372 4549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:03.772195 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 4483
I20260929 20:47:03.772404 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:47:03.773113 388 external_mini_cluster.cc:1428] Setting key 27be6b1c337f5cd43b6a3be79b05dd2b
I20260929 20:47:03.778095 4549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69: Bootstrap starting.
I20260929 20:47:03.781752 4549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:03.782950 4549 log.cc:824] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:03.786024 4549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69: No bootstrap required, opened a new log
Sep 29 20:47:03 dist-test-slave-blv7 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:47:03.794390 4549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdce833cc084f50bc62e60e2cf11b69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } }
I20260929 20:47:03.794893 4549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:03.795038 4549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cdce833cc084f50bc62e60e2cf11b69, State: Initialized, Role: FOLLOWER
I20260929 20:47:03.795588 4549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdce833cc084f50bc62e60e2cf11b69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } }
I20260929 20:47:03.795790 4549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:47:03.795912 4549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:47:03.796056 4549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:03.796172 4552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:03.775057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44400 (local address 127.0.97.62:41411)
0929 20:47:03.775844 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:03.775855 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:03.775894 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:47:03.777510 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:03.777532 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:03.778182 (+ 650us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:03.778644 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:03.780322 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:03.781853 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:03.783610 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:03.784478 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:03.784622 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:03.789033 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:03.789042 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:03.789056 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:03.789091 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:03.792011 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:03.792728 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:03.792734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:03.792740 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:03.792896 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:03.793322 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:03.793326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:03.793328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:03.793890 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:03.794126 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:03.794533 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 20:47:03.795011 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":389,"threads_started":1}
I20260929 20:47:03.798650 4549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdce833cc084f50bc62e60e2cf11b69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } }
I20260929 20:47:03.799176 4549 leader_election.cc:302] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cdce833cc084f50bc62e60e2cf11b69; no voters:
I20260929 20:47:03.799676 4549 leader_election.cc:288] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:47:03.799783 4554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:47:03.800452 4554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69 [term 1 LEADER]: Becoming Leader. State: Replica: 3cdce833cc084f50bc62e60e2cf11b69, State: Running, Role: LEADER
I20260929 20:47:03.800957 4554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdce833cc084f50bc62e60e2cf11b69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } }
I20260929 20:47:03.801612 4549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:47:03.804280 4555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cdce833cc084f50bc62e60e2cf11b69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdce833cc084f50bc62e60e2cf11b69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } } }
I20260929 20:47:03.804584 4555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69 [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:03.807211 4556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cdce833cc084f50bc62e60e2cf11b69. Latest consensus state: current_term: 1 leader_uuid: "3cdce833cc084f50bc62e60e2cf11b69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cdce833cc084f50bc62e60e2cf11b69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41411 } } }
I20260929 20:47:03.807518 4556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69 [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:03.808032 4562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:47:03.812420 4562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:47:03.820794 4562 catalog_manager.cc:1347] Generated new cluster ID: 956f9b282c2b4a8eae2eeb800d97a02c
I20260929 20:47:03.820909 4562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:47:03.841077 4562 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:47:03.842847 4562 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:47:03.856017 4562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69: Generated new TSK 0
I20260929 20:47:03.856781 4562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:47:03.898340 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41411
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:04.202589 4575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:04.202911 4575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:04.203091 4575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:04.210197 4575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:04.210299 4575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:04.210466 4575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:47:04.219991 4575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:04.221230 4575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:04.222471 4575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:04.234423 4580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:04.234867 4581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:04.234889 4583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:04.235256 4575 server_base.cc:1034] running on GCE node
I20260929 20:47:04.235756 4575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:04.236593 4575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:04.237807 4575 hybrid_clock.cc:648] HybridClock initialized: now 1790714824237760 us; error 34 us; skew 500 ppm
I20260929 20:47:04.238118 4575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:04 dist-test-slave-blv7 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:04.241029 4575 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:47:04.242043 4575 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:04.242126 4575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:04.242411 4575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:04.245177 4575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "b830168b87d143df82f67df68cb10665"
format_stamp: "Formatted at 2026-09-29 20:47:04 on dist-test-slave-blv7"
server_key: "6f14d614966003b3de528065396bffe7"
server_key_iv: "35214586bed2a7c3b8f6e2dd755da222"
server_key_version: "encryptionkey@0"
I20260929 20:47:04.245914 4575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b830168b87d143df82f67df68cb10665"
format_stamp: "Formatted at 2026-09-29 20:47:04 on dist-test-slave-blv7"
server_key: "6f14d614966003b3de528065396bffe7"
server_key_iv: "35214586bed2a7c3b8f6e2dd755da222"
server_key_version: "encryptionkey@0"
I20260929 20:47:04.250414 4575 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 20:47:04.253654 4589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:04.254856 4575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:47:04.254987 4575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "b830168b87d143df82f67df68cb10665"
format_stamp: "Formatted at 2026-09-29 20:47:04 on dist-test-slave-blv7"
server_key: "6f14d614966003b3de528065396bffe7"
server_key_iv: "35214586bed2a7c3b8f6e2dd755da222"
server_key_version: "encryptionkey@0"
I20260929 20:47:04.255232 4575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:04.278865 4575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:04.282480 4575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:04.282722 4575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:04.283658 4575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:04.284925 4575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:04.285010 4575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:04.285109 4575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:04.285180 4575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:04.314096 4575 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32777
I20260929 20:47:04.314162 4701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32777 every 8 connection(s)
I20260929 20:47:04.314891 4575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:47:04.324410 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 4575
I20260929 20:47:04.324605 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:47:04.325181 388 external_mini_cluster.cc:1428] Setting key 453efc3ebc4a2999f478aa4f1341d5cd
Sep 29 20:47:04 dist-test-slave-blv7 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:47:04.333343 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:04.317160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53115 (local address 127.0.97.62:41411)
0929 20:47:04.317547 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:04.317551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:04.318569 (+ 1018us) server_negotiation.cc:407] Connection header received
0929 20:47:04.320336 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:04.320341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:04.320418 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:04.320512 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:04.321947 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:04.322714 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:04.324277 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:04.324712 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:04.324830 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:04.328163 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:04.328168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:04.328171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:04.328210 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:04.330567 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:04.331278 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:04.331283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:04.331286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:04.331392 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:04.331774 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:04.331779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:04.331782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:04.332031 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:04.332154 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:04.332791 (+ 637us) server_negotiation.cc:299] Negotiation successful
0929 20:47:04.332988 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":195,"threads_started":1}
I20260929 20:47:04.334287 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:04.317778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.1:53115)
0929 20:47:04.318425 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:04.318468 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:04.320090 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:04.320746 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:04.320754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:04.321247 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:04.321788 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:04.321805 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:04.322850 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:04.322854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:04.324150 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:04.324162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:04.324911 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:04.324918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:04.325145 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:04.325751 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:04.325772 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:04.327935 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:04.330701 (+ 2766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:04.330710 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:04.330726 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:04.331169 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:04.331486 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:04.331491 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:04.331495 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:04.331669 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:04.332149 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:04.332157 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:04.332464 (+ 307us) client_negotiation.cc:769] Sending connection context
0929 20:47:04.332679 (+ 215us) client_negotiation.cc:240] Negotiation successful
0929 20:47:04.333183 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":270,"threads_started":1}
I20260929 20:47:04.335738 4702 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41411
I20260929 20:47:04.336045 4702 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:04.336745 4702 heartbeater.cc:511] Master 127.0.97.62:41411 requested a full tablet report, sending...
I20260929 20:47:04.339066 4514 ts_manager.cc:194] Registered new tserver with Master: b830168b87d143df82f67df68cb10665 (127.0.97.1:32777)
I20260929 20:47:04.341111 4514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53115
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:47:04.365485 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41411
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:04.662086 4708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:04.662338 4708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:04.662475 4708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:04.669387 4708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:04.669480 4708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:04.669628 4708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:47:04.678730 4708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:04.680152 4708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:04.681556 4708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:04.694060 4714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:04.694152 4713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:04.694185 4716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:04.694710 4708 server_base.cc:1034] running on GCE node
I20260929 20:47:04.695307 4708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:04.696135 4708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:04.697341 4708 hybrid_clock.cc:648] HybridClock initialized: now 1790714824697283 us; error 43 us; skew 500 ppm
I20260929 20:47:04.697629 4708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:04 dist-test-slave-blv7 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:04.700548 4708 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:47:04.701653 4708 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:04.701745 4708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:04.702037 4708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:04.704804 4708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "05f0b83f4627421798b0a6031887595a"
format_stamp: "Formatted at 2026-09-29 20:47:04 on dist-test-slave-blv7"
server_key: "8b92fa1c3b42ae42d78bdb3656c48e6b"
server_key_iv: "8348da1e46a99c4d8935d83a2131b602"
server_key_version: "encryptionkey@0"
I20260929 20:47:04.705555 4708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "05f0b83f4627421798b0a6031887595a"
format_stamp: "Formatted at 2026-09-29 20:47:04 on dist-test-slave-blv7"
server_key: "8b92fa1c3b42ae42d78bdb3656c48e6b"
server_key_iv: "8348da1e46a99c4d8935d83a2131b602"
server_key_version: "encryptionkey@0"
I20260929 20:47:04.710116 4708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 20:47:04.713296 4722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:04.714563 4708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:47:04.714702 4708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "05f0b83f4627421798b0a6031887595a"
format_stamp: "Formatted at 2026-09-29 20:47:04 on dist-test-slave-blv7"
server_key: "8b92fa1c3b42ae42d78bdb3656c48e6b"
server_key_iv: "8348da1e46a99c4d8935d83a2131b602"
server_key_version: "encryptionkey@0"
I20260929 20:47:04.714887 4708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:04.737215 4708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:04.741079 4708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:04.741302 4708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:04.742157 4708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:04.743440 4708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:04.743533 4708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:04.743631 4708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:04.743698 4708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:04.773407 4708 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33355
I20260929 20:47:04.773443 4834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33355 every 8 connection(s)
I20260929 20:47:04.774199 4708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:47:04.780077 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 4708
I20260929 20:47:04.780274 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:47:04.780803 388 external_mini_cluster.cc:1428] Setting key a1b8d03611688468fda1f11c7ceea441
Sep 29 20:47:04 dist-test-slave-blv7 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:47:04.794314 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:04.776703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33635 (local address 127.0.97.62:41411)
0929 20:47:04.776897 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:04.776903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:04.778037 (+ 1134us) server_negotiation.cc:407] Connection header received
0929 20:47:04.780385 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:04.780391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:04.780461 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:04.780543 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:04.782454 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:04.783538 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:04.785356 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:04.785746 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:04.785849 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:04.789280 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:04.789284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:04.789288 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:04.789316 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:04.791614 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:04.792342 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:04.792345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:04.792347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:04.792459 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:04.792842 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:04.792845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:04.792847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:04.793057 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:04.793162 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:04.793898 (+ 736us) server_negotiation.cc:299] Negotiation successful
0929 20:47:04.794082 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:47:04.795706 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:04.777141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.2:33635)
0929 20:47:04.777864 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:04.777911 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:04.780101 (+ 2190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:04.780803 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:04.780814 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:04.781539 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:04.782280 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:04.782303 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:04.783715 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:04.783721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:04.785228 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:04.785239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:04.785939 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:04.785945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:04.786153 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:04.786732 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:04.786753 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:04.789088 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:04.791732 (+ 2644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:04.791740 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:04.791757 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:04.792250 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:04.792545 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:04.792549 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:04.792552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:04.792756 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:04.793158 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:04.793165 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:04.793525 (+ 360us) client_negotiation.cc:769] Sending connection context
0929 20:47:04.793817 (+ 292us) client_negotiation.cc:240] Negotiation successful
0929 20:47:04.794315 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":268,"threads_started":1}
I20260929 20:47:04.797122 4835 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41411
I20260929 20:47:04.797446 4835 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:47:04.798174 4835 heartbeater.cc:511] Master 127.0.97.62:41411 requested a full tablet report, sending...
I20260929 20:47:04.799495 4514 ts_manager.cc:194] Registered new tserver with Master: 05f0b83f4627421798b0a6031887595a (127.0.97.2:33355)
I20260929 20:47:04.800361 4514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:47:04.822611 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41411
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:05.122220 4840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:05.122483 4840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:05.122579 4840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:05.129663 4840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:05.129760 4840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:05.129897 4840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:47:05.139297 4840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:05.140477 4840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:05.141733 4840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:05.155297 4845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:05.155292 4848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:05.155325 4846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:05.156168 4840 server_base.cc:1034] running on GCE node
I20260929 20:47:05.156688 4840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:05.157570 4840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:05.158759 4840 hybrid_clock.cc:648] HybridClock initialized: now 1790714825158704 us; error 54 us; skew 500 ppm
I20260929 20:47:05.159058 4840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:05 dist-test-slave-blv7 krb5kdc[4470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:05.161939 4840 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:47:05.162966 4840 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:05.163089 4840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:05.163357 4840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:05.165978 4840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "7e675a4f55ad4a939f693589cc20a3aa"
format_stamp: "Formatted at 2026-09-29 20:47:05 on dist-test-slave-blv7"
server_key: "444fb283bb0d06dd8bec14abda28c6e5"
server_key_iv: "372d61ad936d035d9ae6e7d0337c03c1"
server_key_version: "encryptionkey@0"
I20260929 20:47:05.166697 4840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7e675a4f55ad4a939f693589cc20a3aa"
format_stamp: "Formatted at 2026-09-29 20:47:05 on dist-test-slave-blv7"
server_key: "444fb283bb0d06dd8bec14abda28c6e5"
server_key_iv: "372d61ad936d035d9ae6e7d0337c03c1"
server_key_version: "encryptionkey@0"
I20260929 20:47:05.171226 4840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 20:47:05.174283 4854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:05.175504 4840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 20:47:05.175632 4840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "7e675a4f55ad4a939f693589cc20a3aa"
format_stamp: "Formatted at 2026-09-29 20:47:05 on dist-test-slave-blv7"
server_key: "444fb283bb0d06dd8bec14abda28c6e5"
server_key_iv: "372d61ad936d035d9ae6e7d0337c03c1"
server_key_version: "encryptionkey@0"
I20260929 20:47:05.175796 4840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:05.197714 4840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:05.201294 4840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:05.201494 4840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:05.202363 4840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:05.203521 4840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:05.203612 4840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:05.203685 4840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:05.203735 4840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:05.232689 4840 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38061
I20260929 20:47:05.232738 4966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38061 every 8 connection(s)
I20260929 20:47:05.233443 4840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:47:05.238448 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 4840
I20260929 20:47:05.238629 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:47:05.239156 388 external_mini_cluster.cc:1428] Setting key 6e6598a991272cf7a1c63e81f002eccf
Sep 29 20:47:05 dist-test-slave-blv7 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:47:05.251329 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:05.235542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46517 (local address 127.0.97.62:41411)
0929 20:47:05.235700 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:05.235704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:05.236843 (+ 1139us) server_negotiation.cc:407] Connection header received
0929 20:47:05.238647 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:05.238652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:05.238729 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:05.238827 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:05.240722 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.241533 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.243086 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.243482 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.243597 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:05.246682 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:05.246687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:05.246690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:05.246720 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:05.249043 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:05.249625 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:05.249628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:05.249630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:05.249727 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:05.250055 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:05.250058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:05.250060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:05.250289 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:05.250390 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:05.250911 (+ 521us) server_negotiation.cc:299] Negotiation successful
0929 20:47:05.251096 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:47:05.252348 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:05.236016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41411 (local address 127.0.97.3:46517)
0929 20:47:05.236690 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:05.236727 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:05.238352 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:05.239102 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:05.239113 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:05.239822 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:05.240545 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:05.240569 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.241663 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.241668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:05.242949 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:05.242962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.243685 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.243692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:05.243895 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:05.244462 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:05.244483 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:05.246503 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:05.249159 (+ 2656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:05.249165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:05.249176 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:05.249542 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:05.249796 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:05.249799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:05.249801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:05.249983 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:05.250375 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:05.250379 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:05.250646 (+ 267us) client_negotiation.cc:769] Sending connection context
0929 20:47:05.250849 (+ 203us) client_negotiation.cc:240] Negotiation successful
0929 20:47:05.251292 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":239,"threads_started":1}
I20260929 20:47:05.253604 4967 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41411
I20260929 20:47:05.253925 4967 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:05.254523 4967 heartbeater.cc:511] Master 127.0.97.62:41411 requested a full tablet report, sending...
I20260929 20:47:05.255723 4514 ts_manager.cc:194] Registered new tserver with Master: 7e675a4f55ad4a939f693589cc20a3aa (127.0.97.3:38061)
I20260929 20:47:05.256431 4514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46517
I20260929 20:47:05.265985 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:47:05.281107 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:05.270083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44416 (local address 127.0.97.62:41411)
0929 20:47:05.270225 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:05.270229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:05.270689 (+ 460us) server_negotiation.cc:407] Connection header received
0929 20:47:05.270815 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:05.270818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:05.270871 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:05.270944 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:05.272339 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.273001 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.274205 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.274609 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.274691 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:05.276766 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:05.276770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:05.276773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:05.276800 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:05.279094 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:05.279643 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:05.279646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:05.279649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:05.279754 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:05.280042 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:05.280045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:05.280047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:05.280323 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:05.280486 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:05.280761 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 20:47:05.280900 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 20:47:05.284741 4514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:47:05.291155 4514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:47:05.313175 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:05.304089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33355 (local address 127.0.0.1:35358)
0929 20:47:05.304629 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:05.304653 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:05.304766 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:05.306533 (+ 1767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:05.306538 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:05.306561 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:05.306934 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:05.306946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.308617 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.308622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:05.310556 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:05.310567 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.312292 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.312298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:05.312479 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:05.312532 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 20:47:05.312660 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 20:47:05.312834 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":97,"thread_start_us":279,"threads_started":1}
I20260929 20:47:05.313560 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:05.302959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40218 (local address 127.0.97.3:38061)
0929 20:47:05.303773 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:05.303782 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:05.303815 (+ 33us) server_negotiation.cc:407] Connection header received
0929 20:47:05.304097 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:05.304106 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:05.304384 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:05.304564 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:05.306428 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.307878 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.310838 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.312152 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.312259 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:05.312469 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:05.313084 (+ 615us) server_negotiation.cc:299] Negotiation successful
0929 20:47:05.313247 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":484,"threads_started":1}
I20260929 20:47:05.313958 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:05.304363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.0.97.2:33355)
0929 20:47:05.305966 (+ 1603us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:05.305976 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:05.306005 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:47:05.306099 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:05.306107 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:05.306329 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:05.306538 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:05.307154 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.308458 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.310711 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.312124 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.312255 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:05.312440 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:05.313503 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0929 20:47:05.313675 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1458,"thread_start_us":1323,"threads_started":1}
I20260929 20:47:05.314693 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:05.305125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.0.97.1:32777)
0929 20:47:05.306246 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:05.306254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:05.306280 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:47:05.306369 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:05.306376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:05.306584 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:05.306764 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:05.307478 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.308619 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.311283 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.312423 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.312545 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:05.312727 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:05.314339 (+ 1612us) server_negotiation.cc:299] Negotiation successful
0929 20:47:05.314467 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":975,"thread_start_us":885,"threads_started":1}
I20260929 20:47:05.314735 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:05.304681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32777 (local address 127.0.0.1:51048)
0929 20:47:05.305333 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:05.305357 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:05.305490 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:05.306823 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:05.306828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:05.306852 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:05.307299 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:05.307312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.308765 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.308771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:05.311138 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:05.311149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.312644 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.312650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:05.314165 (+ 1515us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:05.314198 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:47:05.314284 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 20:47:05.314401 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":78,"thread_start_us":570,"threads_started":1}
I20260929 20:47:05.315685 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:05.302676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38061 (local address 127.0.0.1:40218)
0929 20:47:05.303693 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:05.303735 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:05.303876 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:05.304700 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:05.304707 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:05.304736 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:05.305160 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:05.305174 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.308012 (+ 2838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.308018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:05.310677 (+ 2659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:05.310692 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.312634 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.312641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:05.312899 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:05.312933 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:47:05.313029 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 20:47:05.313153 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":64,"thread_start_us":592,"threads_started":1}
I20260929 20:47:05.319087 4902 tablet_service.cc:1467] Processing CreateTablet for tablet 3f4a759ace7946be8d8f079d7ccb7ab2 (DEFAULT_TABLE table=test-table [id=d62c6bf03a2047cc98b0b0aa3ef21044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:05.319093 4637 tablet_service.cc:1467] Processing CreateTablet for tablet 3f4a759ace7946be8d8f079d7ccb7ab2 (DEFAULT_TABLE table=test-table [id=d62c6bf03a2047cc98b0b0aa3ef21044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:05.320207 4902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f4a759ace7946be8d8f079d7ccb7ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:05.320326 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f4a759ace7946be8d8f079d7ccb7ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:05.325533 4770 tablet_service.cc:1467] Processing CreateTablet for tablet 3f4a759ace7946be8d8f079d7ccb7ab2 (DEFAULT_TABLE table=test-table [id=d62c6bf03a2047cc98b0b0aa3ef21044]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:05.326785 4770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f4a759ace7946be8d8f079d7ccb7ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:05.330407 4985 tablet_bootstrap.cc:492] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa: Bootstrap starting.
I20260929 20:47:05.332773 4986 tablet_bootstrap.cc:492] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665: Bootstrap starting.
I20260929 20:47:05.335115 4985 tablet_bootstrap.cc:654] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:05.336001 4986 tablet_bootstrap.cc:654] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:05.336313 4985 log.cc:824] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:05.337122 4986 log.cc:824] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:05.338557 4989 tablet_bootstrap.cc:492] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a: Bootstrap starting.
I20260929 20:47:05.339493 4985 tablet_bootstrap.cc:492] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa: No bootstrap required, opened a new log
I20260929 20:47:05.339758 4985 ts_tablet_manager.cc:1397] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260929 20:47:05.341557 4986 tablet_bootstrap.cc:492] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665: No bootstrap required, opened a new log
I20260929 20:47:05.341795 4986 ts_tablet_manager.cc:1397] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 20:47:05.341815 4989 tablet_bootstrap.cc:654] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:05.342922 4989 log.cc:824] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:05.343951 4702 heartbeater.cc:503] Master 127.0.97.62:41411 was elected leader, sending a full tablet report...
I20260929 20:47:05.346751 4989 tablet_bootstrap.cc:492] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a: No bootstrap required, opened a new log
I20260929 20:47:05.347074 4989 ts_tablet_manager.cc:1397] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260929 20:47:05.347735 4985 raft_consensus.cc:348] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e675a4f55ad4a939f693589cc20a3aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38061 } } peers { permanent_uuid: "05f0b83f4627421798b0a6031887595a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } } peers { permanent_uuid: "b830168b87d143df82f67df68cb10665" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32777 } }
I20260929 20:47:05.348312 4985 raft_consensus.cc:374] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:05.348448 4985 raft_consensus.cc:729] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e675a4f55ad4a939f693589cc20a3aa, State: Initialized, Role: FOLLOWER
I20260929 20:47:05.349154 4985 consensus_queue.cc:260] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e675a4f55ad4a939f693589cc20a3aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38061 } } peers { permanent_uuid: "05f0b83f4627421798b0a6031887595a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } } peers { permanent_uuid: "b830168b87d143df82f67df68cb10665" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32777 } }
I20260929 20:47:05.350098 4967 heartbeater.cc:503] Master 127.0.97.62:41411 was elected leader, sending a full tablet report...
I20260929 20:47:05.351493 4986 raft_consensus.cc:348] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e675a4f55ad4a939f693589cc20a3aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38061 } } peers { permanent_uuid: "05f0b83f4627421798b0a6031887595a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } } peers { permanent_uuid: "b830168b87d143df82f67df68cb10665" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32777 } }
I20260929 20:47:05.351933 4986 raft_consensus.cc:374] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:05.352067 4986 raft_consensus.cc:729] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b830168b87d143df82f67df68cb10665, State: Initialized, Role: FOLLOWER
I20260929 20:47:05.352180 4985 ts_tablet_manager.cc:1428] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260929 20:47:05.352735 4986 consensus_queue.cc:260] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e675a4f55ad4a939f693589cc20a3aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38061 } } peers { permanent_uuid: "05f0b83f4627421798b0a6031887595a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } } peers { permanent_uuid: "b830168b87d143df82f67df68cb10665" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32777 } }
I20260929 20:47:05.354404 4986 ts_tablet_manager.cc:1428] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 20:47:05.356139 4989 raft_consensus.cc:348] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e675a4f55ad4a939f693589cc20a3aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38061 } } peers { permanent_uuid: "05f0b83f4627421798b0a6031887595a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } } peers { permanent_uuid: "b830168b87d143df82f67df68cb10665" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32777 } }
I20260929 20:47:05.356432 4989 raft_consensus.cc:374] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:05.356530 4989 raft_consensus.cc:729] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f0b83f4627421798b0a6031887595a, State: Initialized, Role: FOLLOWER
I20260929 20:47:05.357015 4989 consensus_queue.cc:260] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e675a4f55ad4a939f693589cc20a3aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38061 } } peers { permanent_uuid: "05f0b83f4627421798b0a6031887595a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } } peers { permanent_uuid: "b830168b87d143df82f67df68cb10665" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32777 } }
I20260929 20:47:05.357879 4835 heartbeater.cc:503] Master 127.0.97.62:41411 was elected leader, sending a full tablet report...
I20260929 20:47:05.358247 4989 ts_tablet_manager.cc:1428] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
W20260929 20:47:05.484769 4968 tablet.cc:2369] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:47:05.525913 4836 tablet.cc:2369] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:47:05.566970 4703 tablet.cc:2369] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:47:05.766168 4993 raft_consensus.cc:482] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:47:05.766444 4993 raft_consensus.cc:504] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e675a4f55ad4a939f693589cc20a3aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38061 } } peers { permanent_uuid: "05f0b83f4627421798b0a6031887595a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } } peers { permanent_uuid: "b830168b87d143df82f67df68cb10665" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32777 } }
I20260929 20:47:05.768134 4993 leader_election.cc:288] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e675a4f55ad4a939f693589cc20a3aa (127.0.97.3:38061), b830168b87d143df82f67df68cb10665 (127.0.97.1:32777)
I20260929 20:47:05.775631 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:05.768451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49095 (local address 127.0.97.3:38061)
0929 20:47:05.768640 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:05.768647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:05.769190 (+ 543us) server_negotiation.cc:407] Connection header received
0929 20:47:05.769470 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:05.769475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:05.769551 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:05.769625 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:05.770303 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.771372 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.773347 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.774458 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.774552 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:05.774661 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:05.775270 (+ 609us) server_negotiation.cc:299] Negotiation successful
0929 20:47:05.775394 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:47:05.776443 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:05.769428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34649 (local address 127.0.97.1:32777)
0929 20:47:05.769595 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:05.769600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:05.769986 (+ 386us) server_negotiation.cc:407] Connection header received
0929 20:47:05.770172 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:05.770177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:05.770246 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:05.770321 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:05.770759 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.771461 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.773953 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.775185 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.775300 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:05.775397 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:05.776063 (+ 666us) server_negotiation.cc:299] Negotiation successful
0929 20:47:05.776203 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 20:47:05.776486 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:05.768447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38061 (local address 127.0.97.2:49095)
0929 20:47:05.769101 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:05.769119 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:05.769300 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:05.769811 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:05.769814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:05.769841 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:05.770168 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:05.770179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.771487 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.771492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:05.773190 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:05.773203 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.774855 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.774861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:05.775063 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:05.775099 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 20:47:05.776030 (+ 931us) client_negotiation.cc:240] Negotiation successful
0929 20:47:05.776160 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":357,"threads_started":1}
I20260929 20:47:05.777328 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:05.769254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32777 (local address 127.0.97.2:34649)
0929 20:47:05.769872 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:05.769893 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:05.770034 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:05.770330 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:05.770334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:05.770354 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:05.770637 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:05.770644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.771600 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.771606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:05.773795 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:05.773808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.775625 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.775631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:05.775854 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:05.775899 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 20:47:05.776881 (+ 982us) client_negotiation.cc:240] Negotiation successful
0929 20:47:05.777014 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":348,"threads_started":1}
I20260929 20:47:05.777450 4922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4a759ace7946be8d8f079d7ccb7ab2" candidate_uuid: "05f0b83f4627421798b0a6031887595a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e675a4f55ad4a939f693589cc20a3aa" is_pre_election: true
I20260929 20:47:05.777910 4922 raft_consensus.cc:2454] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f0b83f4627421798b0a6031887595a in term 0.
I20260929 20:47:05.778501 4725 leader_election.cc:302] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f0b83f4627421798b0a6031887595a, 7e675a4f55ad4a939f693589cc20a3aa; no voters:
I20260929 20:47:05.778597 4657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4a759ace7946be8d8f079d7ccb7ab2" candidate_uuid: "05f0b83f4627421798b0a6031887595a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b830168b87d143df82f67df68cb10665" is_pre_election: true
I20260929 20:47:05.778959 4993 raft_consensus.cc:2781] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:47:05.779109 4993 raft_consensus.cc:482] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:47:05.779119 4657 raft_consensus.cc:2454] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f0b83f4627421798b0a6031887595a in term 0.
I20260929 20:47:05.779217 4993 raft_consensus.cc:3037] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:05.779421 4992 raft_consensus.cc:482] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:47:05.779637 4992 raft_consensus.cc:504] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e675a4f55ad4a939f693589cc20a3aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38061 } } peers { permanent_uuid: "05f0b83f4627421798b0a6031887595a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } } peers { permanent_uuid: "b830168b87d143df82f67df68cb10665" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32777 } }
I20260929 20:47:05.781287 4992 leader_election.cc:288] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e675a4f55ad4a939f693589cc20a3aa (127.0.97.3:38061), 05f0b83f4627421798b0a6031887595a (127.0.97.2:33355)
I20260929 20:47:05.782310 4993 raft_consensus.cc:504] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e675a4f55ad4a939f693589cc20a3aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38061 } } peers { permanent_uuid: "05f0b83f4627421798b0a6031887595a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } } peers { permanent_uuid: "b830168b87d143df82f67df68cb10665" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32777 } }
I20260929 20:47:05.783386 4922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4a759ace7946be8d8f079d7ccb7ab2" candidate_uuid: "05f0b83f4627421798b0a6031887595a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e675a4f55ad4a939f693589cc20a3aa"
I20260929 20:47:05.783612 4922 raft_consensus.cc:3037] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:05.786738 4922 raft_consensus.cc:2454] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f0b83f4627421798b0a6031887595a in term 1.
I20260929 20:47:05.787199 4725 leader_election.cc:302] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f0b83f4627421798b0a6031887595a, 7e675a4f55ad4a939f693589cc20a3aa; no voters:
I20260929 20:47:05.788005 4993 leader_election.cc:288] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a [CANDIDATE]: Term 1 election: Requested vote from peers 7e675a4f55ad4a939f693589cc20a3aa (127.0.97.3:38061), b830168b87d143df82f67df68cb10665 (127.0.97.1:32777)
I20260929 20:47:05.788482 4993 raft_consensus.cc:2781] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:47:05.788851 4993 raft_consensus.cc:686] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a [term 1 LEADER]: Becoming Leader. State: Replica: 05f0b83f4627421798b0a6031887595a, State: Running, Role: LEADER
I20260929 20:47:05.789243 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:05.781823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56441 (local address 127.0.97.3:38061)
0929 20:47:05.781969 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:05.781974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:05.782483 (+ 509us) server_negotiation.cc:407] Connection header received
0929 20:47:05.782725 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:05.782730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:05.782795 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:05.782845 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:05.783522 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.784699 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.787008 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.788147 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.788240 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:05.788333 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:05.788890 (+ 557us) server_negotiation.cc:299] Negotiation successful
0929 20:47:05.789016 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 20:47:05.789870 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:05.781713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38061 (local address 127.0.97.1:56441)
0929 20:47:05.782397 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:05.782415 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:05.782586 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:05.782997 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:05.783002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:05.783055 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:05.783388 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:05.783397 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.784876 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.784881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:05.786843 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:05.786857 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.788523 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.788529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:05.788711 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:05.788744 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:47:05.789427 (+ 683us) client_negotiation.cc:240] Negotiation successful
0929 20:47:05.789555 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":383,"threads_started":1}
I20260929 20:47:05.789925 4993 consensus_queue.cc:237] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e675a4f55ad4a939f693589cc20a3aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38061 } } peers { permanent_uuid: "05f0b83f4627421798b0a6031887595a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } } peers { permanent_uuid: "b830168b87d143df82f67df68cb10665" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32777 } }
I20260929 20:47:05.794327 4657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4a759ace7946be8d8f079d7ccb7ab2" candidate_uuid: "05f0b83f4627421798b0a6031887595a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b830168b87d143df82f67df68cb10665"
I20260929 20:47:05.794534 4657 raft_consensus.cc:3037] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:05.796047 4922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4a759ace7946be8d8f079d7ccb7ab2" candidate_uuid: "b830168b87d143df82f67df68cb10665" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e675a4f55ad4a939f693589cc20a3aa" is_pre_election: true
I20260929 20:47:05.796232 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:05.782784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57543 (local address 127.0.97.2:33355)
0929 20:47:05.782945 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:05.782950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:05.790216 (+ 7266us) server_negotiation.cc:407] Connection header received
0929 20:47:05.790406 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:05.790411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:05.790472 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:05.790525 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:05.791009 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.792111 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.794112 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.795286 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.795389 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:05.795473 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:05.795867 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 20:47:05.795980 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 20:47:05.796512 4922 raft_consensus.cc:2379] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b830168b87d143df82f67df68cb10665 in current term 1: Already voted for candidate 05f0b83f4627421798b0a6031887595a in this term.
I20260929 20:47:05.796247 4514 catalog_manager.cc:5495] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f0b83f4627421798b0a6031887595a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "05f0b83f4627421798b0a6031887595a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e675a4f55ad4a939f693589cc20a3aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f0b83f4627421798b0a6031887595a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b830168b87d143df82f67df68cb10665" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32777 } health_report { overall_health: UNKNOWN } } }
I20260929 20:47:05.796787 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:05.782477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33355 (local address 127.0.97.1:57543)
0929 20:47:05.790143 (+ 7666us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:05.790159 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:05.790289 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:05.790704 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:05.790708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:05.790730 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:05.790864 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:05.790873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.792261 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.792266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:05.793925 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:05.793955 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.795490 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.795496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:05.795695 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:05.795731 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 20:47:05.796368 (+ 637us) client_negotiation.cc:240] Negotiation successful
0929 20:47:05.796506 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7579,"thread_start_us":11345,"threads_started":1}
I20260929 20:47:05.797514 4657 raft_consensus.cc:2454] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f0b83f4627421798b0a6031887595a in term 1.
I20260929 20:47:05.797912 4790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4a759ace7946be8d8f079d7ccb7ab2" candidate_uuid: "b830168b87d143df82f67df68cb10665" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f0b83f4627421798b0a6031887595a" is_pre_election: true
I20260929 20:47:05.798792 4592 leader_election.cc:302] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b830168b87d143df82f67df68cb10665; no voters: 05f0b83f4627421798b0a6031887595a, 7e675a4f55ad4a939f693589cc20a3aa
I20260929 20:47:05.799173 4992 raft_consensus.cc:2726] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 20:47:05.872885 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:05.867871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.0.97.2:33355)
0929 20:47:05.868112 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:05.868118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:05.868407 (+ 289us) server_negotiation.cc:407] Connection header received
0929 20:47:05.868586 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:05.868590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:05.868666 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:05.868776 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:47:05.869163 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.869863 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.871116 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.871477 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.871560 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:05.871941 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:47:05.872458 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:47:05.872559 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:05.872613 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 20:47:05.872674 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 20:47:05.879930 4922 raft_consensus.cc:1261] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa [term 1 FOLLOWER]: Refusing update from remote peer 05f0b83f4627421798b0a6031887595a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:47:05.879949 4657 raft_consensus.cc:1261] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665 [term 1 FOLLOWER]: Refusing update from remote peer 05f0b83f4627421798b0a6031887595a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:47:05.880661 4998 consensus_queue.cc:1035] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e675a4f55ad4a939f693589cc20a3aa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:47:05.881003 4993 consensus_queue.cc:1035] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b830168b87d143df82f67df68cb10665" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:47:05.890095 5002 mvcc.cc:205] Tried to move back new op lower bound from 7334767926795239424 to 7334767926443380736. Current Snapshot: MvccSnapshot[applied={T|T < 7334767926795239424}]
I20260929 20:47:05.891208 5004 mvcc.cc:205] Tried to move back new op lower bound from 7334767926795239424 to 7334767926443380736. Current Snapshot: MvccSnapshot[applied={T|T < 7334767926795239424}]
I20260929 20:47:05.903563 4514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:47:05.903762 4514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:47:05.906618 4514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3cdce833cc084f50bc62e60e2cf11b69: Sending DeleteTablet for 3 replicas of tablet 3f4a759ace7946be8d8f079d7ccb7ab2
I20260929 20:47:05.907536 4902 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f4a759ace7946be8d8f079d7ccb7ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40218
I20260929 20:47:05.907729 4637 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f4a759ace7946be8d8f079d7ccb7ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51048
I20260929 20:47:05.907734 4770 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f4a759ace7946be8d8f079d7ccb7ab2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:05 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35358
I20260929 20:47:05.909433 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 4575
I20260929 20:47:05.920027 5013 tablet_replica.cc:330] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa: stopping tablet replica
I20260929 20:47:05.920589 5013 raft_consensus.cc:2229] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:47:05.921195 5013 raft_consensus.cc:2258] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:05.923131 5013 ts_tablet_manager.cc:1905] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:05.929881 5013 ts_tablet_manager.cc:1918] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:47:05.930090 5013 log.cc:1196] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/3f4a759ace7946be8d8f079d7ccb7ab2
I20260929 20:47:05.930601 5013 ts_tablet_manager.cc:1939] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 7e675a4f55ad4a939f693589cc20a3aa: Deleting consensus metadata
I20260929 20:47:05.932410 4500 catalog_manager.cc:4841] TS 7e675a4f55ad4a939f693589cc20a3aa (127.0.97.3:38061): tablet 3f4a759ace7946be8d8f079d7ccb7ab2 (table test-table [id=d62c6bf03a2047cc98b0b0aa3ef21044]) successfully deleted
I20260929 20:47:05.935997 5014 tablet_replica.cc:330] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665: stopping tablet replica
I20260929 20:47:05.936545 5014 raft_consensus.cc:2229] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:47:05.937166 5014 raft_consensus.cc:2258] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:05.938988 5014 ts_tablet_manager.cc:1905] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:05.943552 5015 tablet_replica.cc:330] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a: stopping tablet replica
I20260929 20:47:05.943962 5015 raft_consensus.cc:2229] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:47:05.944569 5014 ts_tablet_manager.cc:1918] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:47:05.944607 5015 raft_consensus.cc:2258] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:05.944718 5014 log.cc:1196] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/3f4a759ace7946be8d8f079d7ccb7ab2
I20260929 20:47:05.945199 5014 ts_tablet_manager.cc:1939] T 3f4a759ace7946be8d8f079d7ccb7ab2 P b830168b87d143df82f67df68cb10665: Deleting consensus metadata
I20260929 20:47:05.946250 5015 ts_tablet_manager.cc:1905] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:05.946983 4500 catalog_manager.cc:4841] TS b830168b87d143df82f67df68cb10665 (127.0.97.1:32777): tablet 3f4a759ace7946be8d8f079d7ccb7ab2 (table test-table [id=d62c6bf03a2047cc98b0b0aa3ef21044]) successfully deleted
Sep 29 20:47:05 dist-test-slave-blv7 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:47:05.951651 5015 ts_tablet_manager.cc:1918] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:47:05.951793 5015 log.cc:1196] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/3f4a759ace7946be8d8f079d7ccb7ab2
I20260929 20:47:05.952217 5015 ts_tablet_manager.cc:1939] T 3f4a759ace7946be8d8f079d7ccb7ab2 P 05f0b83f4627421798b0a6031887595a: Deleting consensus metadata
I20260929 20:47:05.953501 4500 catalog_manager.cc:4841] TS 05f0b83f4627421798b0a6031887595a (127.0.97.2:33355): tablet 3f4a759ace7946be8d8f079d7ccb7ab2 (table test-table [id=d62c6bf03a2047cc98b0b0aa3ef21044]) successfully deleted
I20260929 20:47:05.957007 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:05.910731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51060 (local address 127.0.97.1:32777)
0929 20:47:05.910903 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:05.910910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:05.934552 (+ 23642us) server_negotiation.cc:407] Connection header received
0929 20:47:05.934712 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:05.934717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:05.934810 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:05.934906 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:05.936558 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.937645 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.939353 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:05.939949 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:05.940040 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:05.950830 (+ 10790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:05.950843 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:05.950860 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:05.950902 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:05.954415 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:05.955009 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:05.955040 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:05.955044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:05.955191 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:05.955506 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:05.955510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:05.955514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:05.955969 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:05.956248 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:05.956430 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 20:47:05.956645 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 20:47:05.957583 4697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:06.098407 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 4575
I20260929 20:47:06.112198 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 4708
Sep 29 20:47:06 dist-test-slave-blv7 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:47:06.128270 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:06.112908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35372 (local address 127.0.97.2:33355)
0929 20:47:06.113101 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:06.113107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:06.113128 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:47:06.113246 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:06.113252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:06.113350 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:06.113461 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:06.114668 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:06.115777 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:06.117023 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:06.117689 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:06.117809 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:06.122042 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:06.122058 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:06.122084 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:06.122121 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:06.125699 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:06.126272 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:06.126281 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:06.126285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:06.126458 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:06.126783 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:06.126788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:06.126791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:06.127241 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:06.127520 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:06.127706 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 20:47:06.127905 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:47:06.128979 4830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:06.237545 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 4708
I20260929 20:47:06.255419 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 4840
Sep 29 20:47:06 dist-test-slave-blv7 krb5kdc[4470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:47:06.268949 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:06.256053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40220 (local address 127.0.97.3:38061)
0929 20:47:06.256221 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:06.256227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:06.256270 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:47:06.256461 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:06.256465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:06.256521 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:06.256597 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:06.257780 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:06.258469 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:06.259719 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:06.260117 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:06.260219 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:06.264033 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:06.264045 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:06.264060 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:06.264095 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:06.266507 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:06.267125 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:06.267135 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:06.267137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:06.267253 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:06.267590 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:06.267595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:06.267597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:06.267907 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:06.268167 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:06.268463 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 20:47:06.268666 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:47:06.269529 4962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:06.354863 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 4840
I20260929 20:47:06.367444 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 4483
I20260929 20:47:06.368486 4544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:06.472114 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 4483
2026-09-29T20:47:06Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4262 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T20:47:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:47:06Z Disabled control of system clock
I20260929 20:47:06.528182 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44991
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 20:47:06.828693 5024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:06.828991 5024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:06.829093 5024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:06.836222 5024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:47:06.836323 5024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:06.836371 5024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:06.836432 5024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:47:06.836483 5024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:47:06.845441 5024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:06.846599 5024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:06.848398 5024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:06.859589 5029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:06.859632 5030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:06.859546 5032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:06.860483 5024 server_base.cc:1034] running on GCE node
I20260929 20:47:06.861188 5024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:06.862433 5024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:06.863667 5024 hybrid_clock.cc:648] HybridClock initialized: now 1790714826863619 us; error 37 us; skew 500 ppm
I20260929 20:47:06.863982 5024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:47:06.865967 5024 webserver.cc:460] Webserver started at http://127.0.97.62:42349/ using document root <none> and password file <none>
I20260929 20:47:06.866564 5024 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:06.866663 5024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:06.866947 5024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:06.869611 5024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "b75662c478ce414e8e89d6e78b8e6c41"
format_stamp: "Formatted at 2026-09-29 20:47:06 on dist-test-slave-blv7"
server_key: "150ab76fa8ee49d1951f8af21e5d1995"
server_key_iv: "dfbab63aa572216412e617062219c23e"
server_key_version: "encryptionkey@0"
I20260929 20:47:06.870338 5024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "b75662c478ce414e8e89d6e78b8e6c41"
format_stamp: "Formatted at 2026-09-29 20:47:06 on dist-test-slave-blv7"
server_key: "150ab76fa8ee49d1951f8af21e5d1995"
server_key_iv: "dfbab63aa572216412e617062219c23e"
server_key_version: "encryptionkey@0"
I20260929 20:47:06.874935 5024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260929 20:47:06.877961 5038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:06.879194 5024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 20:47:06.879333 5024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "b75662c478ce414e8e89d6e78b8e6c41"
format_stamp: "Formatted at 2026-09-29 20:47:06 on dist-test-slave-blv7"
server_key: "150ab76fa8ee49d1951f8af21e5d1995"
server_key_iv: "dfbab63aa572216412e617062219c23e"
server_key_version: "encryptionkey@0"
I20260929 20:47:06.879508 5024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:06.884673 5024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:06.885330 5024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:06.885574 5024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:06.901235 5024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44991
I20260929 20:47:06.901265 5089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44991 every 8 connection(s)
I20260929 20:47:06.902401 5024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:47:06.905717 5090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:06.912456 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 5024
I20260929 20:47:06.912627 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:47:06.913395 388 external_mini_cluster.cc:1428] Setting key 3f209d4582c463fbbf35a0d8347733bf
I20260929 20:47:06.916013 5090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41: Bootstrap starting.
I20260929 20:47:06.919179 5090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:06.920383 5090 log.cc:824] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:06.921209 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:06.915305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38286 (local address 127.0.97.62:44991)
0929 20:47:06.916225 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:06.916235 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:06.916275 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:47:06.917827 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:06.917848 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:06.918289 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:06.918585 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:06.919683 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:06.919690 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:06.919702 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:06.919717 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:06.919771 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:06.919786 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:06.919923 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:06.920059 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 20:47:06.920288 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":344,"threads_started":1}
I20260929 20:47:06.922947 5090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41: No bootstrap required, opened a new log
I20260929 20:47:06.929837 5090 raft_consensus.cc:348] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75662c478ce414e8e89d6e78b8e6c41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44991 } }
I20260929 20:47:06.930161 5090 raft_consensus.cc:374] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:06.930233 5090 raft_consensus.cc:729] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75662c478ce414e8e89d6e78b8e6c41, State: Initialized, Role: FOLLOWER
I20260929 20:47:06.930817 5090 consensus_queue.cc:260] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75662c478ce414e8e89d6e78b8e6c41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44991 } }
I20260929 20:47:06.931092 5090 raft_consensus.cc:388] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:47:06.931217 5090 raft_consensus.cc:482] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:47:06.931363 5090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:06.933372 5090 raft_consensus.cc:504] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75662c478ce414e8e89d6e78b8e6c41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44991 } }
I20260929 20:47:06.933815 5090 leader_election.cc:302] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b75662c478ce414e8e89d6e78b8e6c41; no voters:
I20260929 20:47:06.934337 5090 leader_election.cc:288] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:47:06.934442 5095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:47:06.935122 5095 raft_consensus.cc:686] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41 [term 1 LEADER]: Becoming Leader. State: Replica: b75662c478ce414e8e89d6e78b8e6c41, State: Running, Role: LEADER
I20260929 20:47:06.935647 5095 consensus_queue.cc:237] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75662c478ce414e8e89d6e78b8e6c41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44991 } }
I20260929 20:47:06.936260 5090 sys_catalog.cc:564] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:47:06.938273 5096 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b75662c478ce414e8e89d6e78b8e6c41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75662c478ce414e8e89d6e78b8e6c41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44991 } } }
I20260929 20:47:06.938616 5096 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41 [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:06.939291 5097 sys_catalog.cc:455] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b75662c478ce414e8e89d6e78b8e6c41. Latest consensus state: current_term: 1 leader_uuid: "b75662c478ce414e8e89d6e78b8e6c41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b75662c478ce414e8e89d6e78b8e6c41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44991 } } }
I20260929 20:47:06.939667 5097 sys_catalog.cc:458] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41 [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:06.940548 5103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:47:06.944651 5103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:47:06.952409 5103 catalog_manager.cc:1347] Generated new cluster ID: 8c49c44479424e57b987531f519a3315
I20260929 20:47:06.952512 5103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:47:06.962546 5103 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:47:06.962882 5103 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:47:06.970201 5103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41: Generated new TSK 0
I20260929 20:47:06.970863 5103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:47:06.981956 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44991
--builtin_ntp_servers=127.0.97.20:44343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 20:47:07.287549 5114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:07.287801 5114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:07.287909 5114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:07.294863 5114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:07.294953 5114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:07.295101 5114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:47:07.303831 5114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:07.305002 5114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:07.306766 5114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:07.318830 5119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:07.318805 5122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:07.319777 5120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:07.320123 5114 server_base.cc:1034] running on GCE node
I20260929 20:47:07.320650 5114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:07.321527 5114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:07.322736 5114 hybrid_clock.cc:648] HybridClock initialized: now 1790714827322690 us; error 34 us; skew 500 ppm
I20260929 20:47:07.323051 5114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:47:07.325182 5114 webserver.cc:460] Webserver started at http://127.0.97.1:45445/ using document root <none> and password file <none>
I20260929 20:47:07.325814 5114 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:07.325907 5114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:07.326184 5114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:07.328913 5114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "739396bb72454fe1870977eb65bb1e5a"
format_stamp: "Formatted at 2026-09-29 20:47:07 on dist-test-slave-blv7"
server_key: "f97a6ca23ccf6b2e535565d7094f17e2"
server_key_iv: "28d07f50bf3f882ca6c61758bdf8f58a"
server_key_version: "encryptionkey@0"
I20260929 20:47:07.329641 5114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "739396bb72454fe1870977eb65bb1e5a"
format_stamp: "Formatted at 2026-09-29 20:47:07 on dist-test-slave-blv7"
server_key: "f97a6ca23ccf6b2e535565d7094f17e2"
server_key_iv: "28d07f50bf3f882ca6c61758bdf8f58a"
server_key_version: "encryptionkey@0"
I20260929 20:47:07.334388 5114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260929 20:47:07.337843 5128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:07.339112 5114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:47:07.339247 5114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "739396bb72454fe1870977eb65bb1e5a"
format_stamp: "Formatted at 2026-09-29 20:47:07 on dist-test-slave-blv7"
server_key: "f97a6ca23ccf6b2e535565d7094f17e2"
server_key_iv: "28d07f50bf3f882ca6c61758bdf8f58a"
server_key_version: "encryptionkey@0"
I20260929 20:47:07.339442 5114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:07.344604 5114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:07.345297 5114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:07.345496 5114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:07.346280 5114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:07.347558 5114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:07.347638 5114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:07.347723 5114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:07.347774 5114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:07.376506 5114 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42213
I20260929 20:47:07.376555 5240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42213 every 8 connection(s)
I20260929 20:47:07.377777 5114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:47:07.385331 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:07.380160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45729 (local address 127.0.97.62:44991)
0929 20:47:07.380311 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:07.380315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:07.381707 (+ 1392us) server_negotiation.cc:407] Connection header received
0929 20:47:07.383412 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:07.383416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:07.383467 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:07.383543 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:07.384777 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:07.384782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:07.384787 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:07.384803 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:07.384832 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:07.384843 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:07.384939 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:07.385079 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 20:47:07.385124 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:47:07.386286 5243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:07.380632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44991 (local address 127.0.97.1:45729)
0929 20:47:07.381562 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:07.381603 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:07.383162 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:07.383796 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:07.383805 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:07.383898 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:07.384493 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:47:07.384496 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:07.384532 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:47:07.384619 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:07.384916 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:07.384921 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:07.384927 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 20:47:07.385139 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 20:47:07.385330 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":266,"threads_started":1}
I20260929 20:47:07.387692 5241 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44991
I20260929 20:47:07.387970 5241 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:07.388057 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 5114
I20260929 20:47:07.388233 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:47:07.388654 5241 heartbeater.cc:511] Master 127.0.97.62:44991 requested a full tablet report, sending...
I20260929 20:47:07.388859 388 external_mini_cluster.cc:1428] Setting key d350468816e54104797f4ffd23653dc8
I20260929 20:47:07.390744 5055 ts_manager.cc:194] Registered new tserver with Master: 739396bb72454fe1870977eb65bb1e5a (127.0.97.1:42213)
I20260929 20:47:07.392007 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44991
--builtin_ntp_servers=127.0.97.20:44343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 20:47:07.691051 5244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:07.691365 5244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:07.691547 5244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:07.698781 5244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:07.698894 5244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:07.699083 5244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:47:07.708155 5244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:07.709461 5244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:07.711270 5244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:07.723604 5250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:07.723650 5252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:07.723670 5249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:07.728513 5244 server_base.cc:1034] running on GCE node
I20260929 20:47:07.729053 5244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:07.729924 5244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:07.731112 5244 hybrid_clock.cc:648] HybridClock initialized: now 1790714827731056 us; error 40 us; skew 500 ppm
I20260929 20:47:07.731393 5244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:47:07.733420 5244 webserver.cc:460] Webserver started at http://127.0.97.2:33391/ using document root <none> and password file <none>
I20260929 20:47:07.734043 5244 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:07.734153 5244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:07.734439 5244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:07.737331 5244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "cee9b02b85af478fa1d5fe422edadef1"
format_stamp: "Formatted at 2026-09-29 20:47:07 on dist-test-slave-blv7"
server_key: "0e04ec7ce4091bfa84f265b183dd275f"
server_key_iv: "1b1a209b825a75dd7cf93540fa179f82"
server_key_version: "encryptionkey@0"
I20260929 20:47:07.738046 5244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "cee9b02b85af478fa1d5fe422edadef1"
format_stamp: "Formatted at 2026-09-29 20:47:07 on dist-test-slave-blv7"
server_key: "0e04ec7ce4091bfa84f265b183dd275f"
server_key_iv: "1b1a209b825a75dd7cf93540fa179f82"
server_key_version: "encryptionkey@0"
I20260929 20:47:07.742856 5244 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 20:47:07.746240 5258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:07.747539 5244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:47:07.747677 5244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "cee9b02b85af478fa1d5fe422edadef1"
format_stamp: "Formatted at 2026-09-29 20:47:07 on dist-test-slave-blv7"
server_key: "0e04ec7ce4091bfa84f265b183dd275f"
server_key_iv: "1b1a209b825a75dd7cf93540fa179f82"
server_key_version: "encryptionkey@0"
I20260929 20:47:07.747867 5244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:07.753052 5244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:07.753756 5244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:07.753979 5244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:07.754813 5244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:07.756145 5244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:07.756224 5244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:07.756320 5244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:07.756379 5244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:07.784984 5244 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43767
I20260929 20:47:07.785061 5370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43767 every 8 connection(s)
I20260929 20:47:07.786199 5244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:47:07.787165 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 5244
I20260929 20:47:07.787333 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:47:07.787881 388 external_mini_cluster.cc:1428] Setting key 242ec656ce2331d0aed84f9ba9f70d75
I20260929 20:47:07.790751 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44991
--builtin_ntp_servers=127.0.97.20:44343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 20:47:07.795166 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:07.788901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34951 (local address 127.0.97.62:44991)
0929 20:47:07.789064 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:07.789070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:07.790480 (+ 1410us) server_negotiation.cc:407] Connection header received
0929 20:47:07.792492 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:07.792498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:07.792580 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:07.792685 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:07.794337 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:07.794344 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:07.794349 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:07.794374 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:07.794414 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:07.794443 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:07.794578 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:07.794758 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 20:47:07.794822 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 20:47:07.796257 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:07.789426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44991 (local address 127.0.97.2:34951)
0929 20:47:07.790344 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:07.790380 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:07.792166 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:07.793035 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:07.793045 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:07.793175 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:07.793920 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:47:07.793924 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:07.793975 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:47:07.794078 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:07.794553 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:07.794560 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:07.794567 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:47:07.794817 (+ 250us) client_negotiation.cc:240] Negotiation successful
0929 20:47:07.795078 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":295,"threads_started":1}
I20260929 20:47:07.797456 5371 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44991
I20260929 20:47:07.797763 5371 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:07.798372 5371 heartbeater.cc:511] Master 127.0.97.62:44991 requested a full tablet report, sending...
I20260929 20:47:07.799744 5055 ts_manager.cc:194] Registered new tserver with Master: cee9b02b85af478fa1d5fe422edadef1 (127.0.97.2:43767)
W20260929 20:47:08.095408 5374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:08.095672 5374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:08.095805 5374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:08.102674 5374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:08.102769 5374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:08.102916 5374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:47:08.111620 5374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:08.112771 5374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:08.114504 5374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:08.126564 5380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:08.126781 5379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:08.127063 5382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:08.127647 5374 server_base.cc:1034] running on GCE node
I20260929 20:47:08.128187 5374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:08.129036 5374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:08.130240 5374 hybrid_clock.cc:648] HybridClock initialized: now 1790714828130197 us; error 30 us; skew 500 ppm
I20260929 20:47:08.130537 5374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:47:08.132797 5374 webserver.cc:460] Webserver started at http://127.0.97.3:44311/ using document root <none> and password file <none>
I20260929 20:47:08.133424 5374 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:08.133522 5374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:08.133802 5374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:08.136513 5374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "bb9b15507ad94ddc86cf9e086c93ac9f"
format_stamp: "Formatted at 2026-09-29 20:47:08 on dist-test-slave-blv7"
server_key: "86ed351c6460f4b73efcdfbf8c4eebaf"
server_key_iv: "34abae89b0805eb36025a7b4fcab2727"
server_key_version: "encryptionkey@0"
I20260929 20:47:08.137252 5374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "bb9b15507ad94ddc86cf9e086c93ac9f"
format_stamp: "Formatted at 2026-09-29 20:47:08 on dist-test-slave-blv7"
server_key: "86ed351c6460f4b73efcdfbf8c4eebaf"
server_key_iv: "34abae89b0805eb36025a7b4fcab2727"
server_key_version: "encryptionkey@0"
I20260929 20:47:08.142196 5374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:47:08.145783 5388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:08.147167 5374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 20:47:08.147317 5374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "bb9b15507ad94ddc86cf9e086c93ac9f"
format_stamp: "Formatted at 2026-09-29 20:47:08 on dist-test-slave-blv7"
server_key: "86ed351c6460f4b73efcdfbf8c4eebaf"
server_key_iv: "34abae89b0805eb36025a7b4fcab2727"
server_key_version: "encryptionkey@0"
I20260929 20:47:08.147524 5374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:08.152825 5374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:08.153566 5374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:08.153798 5374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:08.154634 5374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:08.155933 5374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:08.156033 5374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:08.156134 5374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:08.156203 5374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:08.183727 5374 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39947
I20260929 20:47:08.183779 5500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39947 every 8 connection(s)
I20260929 20:47:08.184945 5374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:47:08.188673 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 5374
I20260929 20:47:08.188827 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:47:08.189224 388 external_mini_cluster.cc:1428] Setting key acc71f364e4ade9d14d6f595a664c185
I20260929 20:47:08.192730 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.187245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46787 (local address 127.0.97.62:44991)
0929 20:47:08.187415 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:08.187419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:08.188827 (+ 1408us) server_negotiation.cc:407] Connection header received
0929 20:47:08.190643 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:08.190648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:08.190720 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:08.190800 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:08.192225 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:08.192230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:08.192233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:08.192248 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:08.192277 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:08.192289 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:08.192375 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:08.192501 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 20:47:08.192543 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:47:08.193650 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.187734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44991 (local address 127.0.97.3:46787)
0929 20:47:08.188637 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:08.188677 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:08.190395 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:08.191075 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:08.191087 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:08.191209 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:08.191938 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:47:08.191942 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:08.191995 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:47:08.192080 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:08.192361 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:08.192366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:08.192371 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:47:08.192559 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 20:47:08.192741 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":280,"threads_started":1}
I20260929 20:47:08.194799 5501 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44991
I20260929 20:47:08.195040 5501 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:08.195540 5501 heartbeater.cc:511] Master 127.0.97.62:44991 requested a full tablet report, sending...
I20260929 20:47:08.196732 5055 ts_manager.cc:194] Registered new tserver with Master: bb9b15507ad94ddc86cf9e086c93ac9f (127.0.97.3:39947)
I20260929 20:47:08.203826 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:47:08.209846 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.207756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38292 (local address 127.0.97.62:44991)
0929 20:47:08.207943 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:08.207949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:08.208376 (+ 427us) server_negotiation.cc:407] Connection header received
0929 20:47:08.208489 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:08.208492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:08.208551 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:08.208631 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:08.209339 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:08.209343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:08.209346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:08.209360 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:08.209388 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:08.209399 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:08.209490 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:08.209542 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 20:47:08.209583 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:47:08.213376 5055 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:47:08.220129 5055 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:47:08.249119 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.231048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36848 (local address 127.0.97.2:43767)
0929 20:47:08.246365 (+ 15317us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:08.246374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:08.246408 (+ 34us) server_negotiation.cc:407] Connection header received
0929 20:47:08.246472 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:08.246480 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:08.246705 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:08.247515 (+ 810us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:08.248339 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:08.248345 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:08.248355 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:08.248370 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:08.248404 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:08.248416 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:08.248554 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:08.248680 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 20:47:08.248775 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15157,"thread_start_us":15756,"threads_started":1}
I20260929 20:47:08.249275 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.230735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43767 (local address 127.0.0.1:36848)
0929 20:47:08.245179 (+ 14444us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:08.245196 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:08.245278 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:08.247075 (+ 1797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:08.247079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:08.247112 (+ 33us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:08.248045 (+ 933us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:47:08.248048 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:08.248067 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:47:08.248142 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:08.248503 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:08.248512 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:08.248520 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 20:47:08.248744 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 20:47:08.248813 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14338,"thread_start_us":12708,"threads_started":1}
I20260929 20:47:08.249325 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.230616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39947 (local address 127.0.0.1:50676)
0929 20:47:08.244542 (+ 13926us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:08.244583 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:08.244719 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:08.246917 (+ 2198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:08.246925 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:08.247057 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:08.248029 (+ 972us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:47:08.248032 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:08.248067 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:47:08.248142 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:08.248570 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:08.248575 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:08.248581 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 20:47:08.248706 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 20:47:08.248835 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13598,"mutex_wait_us":25,"thread_start_us":14324,"threads_started":1}
I20260929 20:47:08.250154 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.232471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42213 (local address 127.0.0.1:46094)
0929 20:47:08.245682 (+ 13211us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:08.245699 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:08.245781 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:08.247339 (+ 1558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:08.247343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:08.247366 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:08.248820 (+ 1454us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:47:08.248823 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:08.248837 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:47:08.248882 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:08.249314 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:08.249319 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:08.249340 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 20:47:08.249732 (+ 392us) client_negotiation.cc:240] Negotiation successful
0929 20:47:08.249776 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13111,"thread_start_us":12988,"threads_started":1}
I20260929 20:47:08.250313 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.230817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50676 (local address 127.0.97.3:39947)
0929 20:47:08.246400 (+ 15583us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:08.246408 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:08.246442 (+ 34us) server_negotiation.cc:407] Connection header received
0929 20:47:08.246505 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:08.246512 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:08.246713 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:08.247394 (+ 681us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:08.248352 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:08.248361 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:08.248374 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:08.248390 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:08.248436 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:08.248455 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:08.249822 (+ 1367us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:08.249885 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 20:47:08.249980 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15269,"thread_start_us":15120,"threads_started":1}
I20260929 20:47:08.250967 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.233145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46094 (local address 127.0.97.1:42213)
0929 20:47:08.245724 (+ 12579us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:08.245733 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:08.245764 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:47:08.245976 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:08.245986 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:08.246239 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:08.246383 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:08.249037 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:08.249044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:08.249055 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:08.249079 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:08.249128 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:08.249147 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:08.249262 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:08.250597 (+ 1335us) server_negotiation.cc:299] Negotiation successful
0929 20:47:08.250684 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12170,"thread_start_us":12046,"threads_started":1}
I20260929 20:47:08.252972 5306 tablet_service.cc:1467] Processing CreateTablet for tablet ce1f3ef2f33f479aabeb467519152920 (DEFAULT_TABLE table=test-table [id=99241a569033440e918de643a91cf24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:08.253420 5436 tablet_service.cc:1467] Processing CreateTablet for tablet ce1f3ef2f33f479aabeb467519152920 (DEFAULT_TABLE table=test-table [id=99241a569033440e918de643a91cf24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:08.254436 5306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce1f3ef2f33f479aabeb467519152920. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:08.254562 5436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce1f3ef2f33f479aabeb467519152920. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:08.254227 5176 tablet_service.cc:1467] Processing CreateTablet for tablet ce1f3ef2f33f479aabeb467519152920 (DEFAULT_TABLE table=test-table [id=99241a569033440e918de643a91cf24a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:08.256088 5176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce1f3ef2f33f479aabeb467519152920. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:08.265069 5519 tablet_bootstrap.cc:492] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f: Bootstrap starting.
I20260929 20:47:08.267829 5520 tablet_bootstrap.cc:492] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1: Bootstrap starting.
I20260929 20:47:08.268365 5519 tablet_bootstrap.cc:654] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:08.269556 5521 tablet_bootstrap.cc:492] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a: Bootstrap starting.
I20260929 20:47:08.269593 5519 log.cc:824] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:08.271268 5520 tablet_bootstrap.cc:654] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:08.272444 5520 log.cc:824] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:08.272516 5521 tablet_bootstrap.cc:654] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:08.273322 5519 tablet_bootstrap.cc:492] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f: No bootstrap required, opened a new log
I20260929 20:47:08.273556 5519 ts_tablet_manager.cc:1397] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 20:47:08.273525 5521 log.cc:824] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:08.275308 5520 tablet_bootstrap.cc:492] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1: No bootstrap required, opened a new log
I20260929 20:47:08.275543 5520 ts_tablet_manager.cc:1397] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 20:47:08.275720 5521 tablet_bootstrap.cc:492] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a: No bootstrap required, opened a new log
I20260929 20:47:08.275920 5521 ts_tablet_manager.cc:1397] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 20:47:08.282644 5521 raft_consensus.cc:348] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9b15507ad94ddc86cf9e086c93ac9f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39947 } } peers { permanent_uuid: "cee9b02b85af478fa1d5fe422edadef1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43767 } } peers { permanent_uuid: "739396bb72454fe1870977eb65bb1e5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42213 } }
I20260929 20:47:08.283062 5521 raft_consensus.cc:374] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:08.283154 5521 raft_consensus.cc:729] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 739396bb72454fe1870977eb65bb1e5a, State: Initialized, Role: FOLLOWER
I20260929 20:47:08.283344 5519 raft_consensus.cc:348] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9b15507ad94ddc86cf9e086c93ac9f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39947 } } peers { permanent_uuid: "cee9b02b85af478fa1d5fe422edadef1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43767 } } peers { permanent_uuid: "739396bb72454fe1870977eb65bb1e5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42213 } }
I20260929 20:47:08.283743 5519 raft_consensus.cc:374] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:08.283839 5519 raft_consensus.cc:729] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb9b15507ad94ddc86cf9e086c93ac9f, State: Initialized, Role: FOLLOWER
I20260929 20:47:08.283793 5521 consensus_queue.cc:260] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9b15507ad94ddc86cf9e086c93ac9f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39947 } } peers { permanent_uuid: "cee9b02b85af478fa1d5fe422edadef1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43767 } } peers { permanent_uuid: "739396bb72454fe1870977eb65bb1e5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42213 } }
I20260929 20:47:08.284488 5519 consensus_queue.cc:260] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9b15507ad94ddc86cf9e086c93ac9f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39947 } } peers { permanent_uuid: "cee9b02b85af478fa1d5fe422edadef1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43767 } } peers { permanent_uuid: "739396bb72454fe1870977eb65bb1e5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42213 } }
I20260929 20:47:08.284840 5241 heartbeater.cc:503] Master 127.0.97.62:44991 was elected leader, sending a full tablet report...
I20260929 20:47:08.285449 5521 ts_tablet_manager.cc:1428] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a: Time spent starting tablet: real 0.009s user 0.002s sys 0.007s
I20260929 20:47:08.285507 5501 heartbeater.cc:503] Master 127.0.97.62:44991 was elected leader, sending a full tablet report...
I20260929 20:47:08.286089 5519 ts_tablet_manager.cc:1428] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260929 20:47:08.286430 5520 raft_consensus.cc:348] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9b15507ad94ddc86cf9e086c93ac9f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39947 } } peers { permanent_uuid: "cee9b02b85af478fa1d5fe422edadef1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43767 } } peers { permanent_uuid: "739396bb72454fe1870977eb65bb1e5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42213 } }
I20260929 20:47:08.286788 5520 raft_consensus.cc:374] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:08.286880 5520 raft_consensus.cc:729] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee9b02b85af478fa1d5fe422edadef1, State: Initialized, Role: FOLLOWER
I20260929 20:47:08.287366 5520 consensus_queue.cc:260] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9b15507ad94ddc86cf9e086c93ac9f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39947 } } peers { permanent_uuid: "cee9b02b85af478fa1d5fe422edadef1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43767 } } peers { permanent_uuid: "739396bb72454fe1870977eb65bb1e5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42213 } }
I20260929 20:47:08.288286 5371 heartbeater.cc:503] Master 127.0.97.62:44991 was elected leader, sending a full tablet report...
I20260929 20:47:08.288877 5520 ts_tablet_manager.cc:1428] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260929 20:47:08.379863 5242 tablet.cc:2369] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:47:08.436457 5502 tablet.cc:2369] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:47:08.439776 5526 raft_consensus.cc:482] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:47:08.440033 5526 raft_consensus.cc:504] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9b15507ad94ddc86cf9e086c93ac9f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39947 } } peers { permanent_uuid: "cee9b02b85af478fa1d5fe422edadef1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43767 } } peers { permanent_uuid: "739396bb72454fe1870977eb65bb1e5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42213 } }
I20260929 20:47:08.441763 5526 leader_election.cc:288] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cee9b02b85af478fa1d5fe422edadef1 (127.0.97.2:43767), 739396bb72454fe1870977eb65bb1e5a (127.0.97.1:42213)
I20260929 20:47:08.452024 5525 raft_consensus.cc:482] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:47:08.452347 5525 raft_consensus.cc:504] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9b15507ad94ddc86cf9e086c93ac9f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39947 } } peers { permanent_uuid: "cee9b02b85af478fa1d5fe422edadef1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43767 } } peers { permanent_uuid: "739396bb72454fe1870977eb65bb1e5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42213 } }
I20260929 20:47:08.453347 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.442108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43767 (local address 127.0.97.3:46149)
0929 20:47:08.442572 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:08.442590 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:08.442686 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:08.451729 (+ 9043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:08.451734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:08.451761 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:08.452441 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:47:08.452445 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:08.452464 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:47:08.452518 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:08.452926 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:08.452931 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:08.452936 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:47:08.453040 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 20:47:08.453095 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345}
I20260929 20:47:08.454231 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.442932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46149 (local address 127.0.97.2:43767)
0929 20:47:08.451214 (+ 8282us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:08.451221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:08.451248 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:47:08.451315 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:08.451321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:08.451382 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:08.451508 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:08.452674 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:08.452680 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:08.452685 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:08.452706 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:08.452745 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:08.452760 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:08.452879 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:08.453928 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0929 20:47:08.453989 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8119}
I20260929 20:47:08.454336 5525 leader_election.cc:288] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb9b15507ad94ddc86cf9e086c93ac9f (127.0.97.3:39947), cee9b02b85af478fa1d5fe422edadef1 (127.0.97.2:43767)
I20260929 20:47:08.455206 5326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce1f3ef2f33f479aabeb467519152920" candidate_uuid: "bb9b15507ad94ddc86cf9e086c93ac9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cee9b02b85af478fa1d5fe422edadef1" is_pre_election: true
I20260929 20:47:08.455698 5326 raft_consensus.cc:2454] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb9b15507ad94ddc86cf9e086c93ac9f in term 0.
I20260929 20:47:08.455868 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.442472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41229 (local address 127.0.97.1:42213)
0929 20:47:08.442736 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:08.442742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:08.452723 (+ 9981us) server_negotiation.cc:407] Connection header received
0929 20:47:08.452991 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:08.452997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:08.453052 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:08.453111 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:08.455181 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:08.455187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:08.455192 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:08.455209 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:08.455243 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:08.455256 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:08.455334 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:08.455552 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 20:47:08.455608 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260929 20:47:08.456410 5389 leader_election.cc:302] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb9b15507ad94ddc86cf9e086c93ac9f, cee9b02b85af478fa1d5fe422edadef1; no voters:
I20260929 20:47:08.456861 5526 raft_consensus.cc:2781] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:47:08.457088 5526 raft_consensus.cc:482] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:47:08.457259 5526 raft_consensus.cc:3037] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:08.457337 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.442356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42213 (local address 127.0.97.3:41229)
0929 20:47:08.452605 (+ 10249us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:08.452628 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:08.452834 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:08.453274 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:08.453278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:08.453299 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:08.454948 (+ 1649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:47:08.454951 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:08.454967 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:47:08.455007 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:08.455414 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:08.455418 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:08.455422 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 20:47:08.456834 (+ 1412us) client_negotiation.cc:240] Negotiation successful
0929 20:47:08.456886 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10040,"thread_start_us":9950,"threads_started":1}
I20260929 20:47:08.458818 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.454784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56775 (local address 127.0.97.3:39947)
0929 20:47:08.455075 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:08.455081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:08.456557 (+ 1476us) server_negotiation.cc:407] Connection header received
0929 20:47:08.456748 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:08.456753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:08.456804 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:08.456878 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:08.458178 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:08.458183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:08.458188 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:08.458205 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:08.458238 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:08.458252 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:08.458335 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:08.458543 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 20:47:08.458591 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260929 20:47:08.458858 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce1f3ef2f33f479aabeb467519152920" candidate_uuid: "bb9b15507ad94ddc86cf9e086c93ac9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "739396bb72454fe1870977eb65bb1e5a" is_pre_election: true
I20260929 20:47:08.459457 5196 raft_consensus.cc:2454] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb9b15507ad94ddc86cf9e086c93ac9f in term 0.
I20260929 20:47:08.459854 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.454807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39947 (local address 127.0.97.1:56775)
0929 20:47:08.456479 (+ 1672us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:08.456496 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:08.456617 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:08.457088 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:08.457092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:08.457125 (+ 33us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:08.457943 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:47:08.457946 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:08.457961 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:47:08.458005 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:08.458420 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:08.458425 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:08.458429 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 20:47:08.459199 (+ 770us) client_negotiation.cc:240] Negotiation successful
0929 20:47:08.459289 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1549,"thread_start_us":1375,"threads_started":1}
I20260929 20:47:08.460731 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.455390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42591 (local address 127.0.97.2:43767)
0929 20:47:08.455536 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:08.455541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:08.458399 (+ 2858us) server_negotiation.cc:407] Connection header received
0929 20:47:08.458619 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:08.458624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:08.458705 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:08.458771 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:08.460029 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:08.460035 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:08.460038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:08.460056 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:08.460086 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:08.460100 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:08.460184 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:08.460393 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 20:47:08.460446 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:47:08.460834 5530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.454948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43767 (local address 127.0.97.1:42591)
0929 20:47:08.458311 (+ 3363us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:08.458326 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:08.458484 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:08.458948 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:08.458952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:08.458973 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:08.459830 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:47:08.459833 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:08.459851 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:47:08.459891 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:08.460265 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:08.460269 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:08.460273 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 20:47:08.460406 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 20:47:08.460464 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3249,"thread_start_us":3079,"threads_started":1}
I20260929 20:47:08.460700 5526 raft_consensus.cc:504] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9b15507ad94ddc86cf9e086c93ac9f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39947 } } peers { permanent_uuid: "cee9b02b85af478fa1d5fe422edadef1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43767 } } peers { permanent_uuid: "739396bb72454fe1870977eb65bb1e5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42213 } }
I20260929 20:47:08.461045 5456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce1f3ef2f33f479aabeb467519152920" candidate_uuid: "739396bb72454fe1870977eb65bb1e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb9b15507ad94ddc86cf9e086c93ac9f" is_pre_election: true
I20260929 20:47:08.461436 5326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce1f3ef2f33f479aabeb467519152920" candidate_uuid: "739396bb72454fe1870977eb65bb1e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cee9b02b85af478fa1d5fe422edadef1" is_pre_election: true
I20260929 20:47:08.461552 5456 raft_consensus.cc:2379] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 739396bb72454fe1870977eb65bb1e5a in current term 1: Already voted for candidate bb9b15507ad94ddc86cf9e086c93ac9f in this term.
I20260929 20:47:08.461643 5326 raft_consensus.cc:2454] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 739396bb72454fe1870977eb65bb1e5a in term 0.
I20260929 20:47:08.461865 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce1f3ef2f33f479aabeb467519152920" candidate_uuid: "bb9b15507ad94ddc86cf9e086c93ac9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "739396bb72454fe1870977eb65bb1e5a"
I20260929 20:47:08.461823 5325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce1f3ef2f33f479aabeb467519152920" candidate_uuid: "bb9b15507ad94ddc86cf9e086c93ac9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cee9b02b85af478fa1d5fe422edadef1"
I20260929 20:47:08.462028 5196 raft_consensus.cc:3037] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:08.462076 5325 raft_consensus.cc:3037] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:08.464145 5196 raft_consensus.cc:2454] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb9b15507ad94ddc86cf9e086c93ac9f in term 1.
I20260929 20:47:08.464315 5526 leader_election.cc:288] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f [CANDIDATE]: Term 1 election: Requested vote from peers cee9b02b85af478fa1d5fe422edadef1 (127.0.97.2:43767), 739396bb72454fe1870977eb65bb1e5a (127.0.97.1:42213)
I20260929 20:47:08.464392 5325 raft_consensus.cc:2454] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb9b15507ad94ddc86cf9e086c93ac9f in term 1.
I20260929 20:47:08.464480 5392 leader_election.cc:302] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 739396bb72454fe1870977eb65bb1e5a, bb9b15507ad94ddc86cf9e086c93ac9f; no voters:
I20260929 20:47:08.464664 5129 leader_election.cc:302] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 739396bb72454fe1870977eb65bb1e5a, cee9b02b85af478fa1d5fe422edadef1; no voters: bb9b15507ad94ddc86cf9e086c93ac9f
I20260929 20:47:08.464792 5526 raft_consensus.cc:2781] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:47:08.465046 5525 raft_consensus.cc:2741] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260929 20:47:08.465256 5526 raft_consensus.cc:686] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f [term 1 LEADER]: Becoming Leader. State: Replica: bb9b15507ad94ddc86cf9e086c93ac9f, State: Running, Role: LEADER
I20260929 20:47:08.465693 5526 consensus_queue.cc:237] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9b15507ad94ddc86cf9e086c93ac9f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39947 } } peers { permanent_uuid: "cee9b02b85af478fa1d5fe422edadef1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43767 } } peers { permanent_uuid: "739396bb72454fe1870977eb65bb1e5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42213 } }
I20260929 20:47:08.469724 5053 catalog_manager.cc:5495] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f reported cstate change: term changed from 0 to 1, leader changed from <none> to bb9b15507ad94ddc86cf9e086c93ac9f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "bb9b15507ad94ddc86cf9e086c93ac9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb9b15507ad94ddc86cf9e086c93ac9f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cee9b02b85af478fa1d5fe422edadef1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "739396bb72454fe1870977eb65bb1e5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42213 } health_report { overall_health: UNKNOWN } } }
I20260929 20:47:08.532572 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.530616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50682 (local address 127.0.97.3:39947)
0929 20:47:08.530791 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:08.530797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:08.530887 (+ 90us) server_negotiation.cc:407] Connection header received
0929 20:47:08.531075 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:08.531079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:08.531184 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:08.531265 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:08.531975 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:08.531979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:08.531981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:08.531992 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:08.532015 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:08.532025 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:08.532128 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:08.532270 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 20:47:08.532313 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 20:47:08.537999 5372 tablet.cc:2369] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:47:08.540428 5325 raft_consensus.cc:1261] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1 [term 1 FOLLOWER]: Refusing update from remote peer bb9b15507ad94ddc86cf9e086c93ac9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:47:08.540446 5196 raft_consensus.cc:1261] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a [term 1 FOLLOWER]: Refusing update from remote peer bb9b15507ad94ddc86cf9e086c93ac9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:47:08.541282 5531 consensus_queue.cc:1035] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "739396bb72454fe1870977eb65bb1e5a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:47:08.541607 5526 consensus_queue.cc:1035] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cee9b02b85af478fa1d5fe422edadef1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:47:08.565723 5055 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:47:08.566000 5055 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:47:08.569300 5055 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b75662c478ce414e8e89d6e78b8e6c41: Sending DeleteTablet for 3 replicas of tablet ce1f3ef2f33f479aabeb467519152920
I20260929 20:47:08.570351 5436 tablet_service.cc:1514] Processing DeleteTablet for tablet ce1f3ef2f33f479aabeb467519152920 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:08 UTC) from {username='slave'} at 127.0.0.1:50676
I20260929 20:47:08.570423 5306 tablet_service.cc:1514] Processing DeleteTablet for tablet ce1f3ef2f33f479aabeb467519152920 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:08 UTC) from {username='slave'} at 127.0.0.1:36848
I20260929 20:47:08.570809 5176 tablet_service.cc:1514] Processing DeleteTablet for tablet ce1f3ef2f33f479aabeb467519152920 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:08 UTC) from {username='slave'} at 127.0.0.1:46094
I20260929 20:47:08.572671 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 5114
I20260929 20:47:08.582005 5546 tablet_replica.cc:330] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a: stopping tablet replica
I20260929 20:47:08.582585 5546 raft_consensus.cc:2229] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:47:08.583212 5546 raft_consensus.cc:2258] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:08.584018 5545 tablet_replica.cc:330] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1: stopping tablet replica
I20260929 20:47:08.584447 5545 raft_consensus.cc:2229] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:47:08.584951 5545 raft_consensus.cc:2258] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:08.585829 5546 ts_tablet_manager.cc:1905] T ce1f3ef2f33f479aabeb467519152920 P 739396bb72454fe1870977eb65bb1e5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:08.586524 5544 tablet_replica.cc:330] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f: stopping tablet replica
I20260929 20:47:08.587323 5544 raft_consensus.cc:2229] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:47:08.588056 5544 raft_consensus.cc:2258] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:08.588177 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.574016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46108 (local address 127.0.97.1:42213)
0929 20:47:08.574188 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:08.574194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:08.585783 (+ 11589us) server_negotiation.cc:407] Connection header received
0929 20:47:08.586049 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:08.586055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:08.586129 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:08.586228 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:08.587308 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:08.587313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:08.587318 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:08.587336 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:08.587375 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:08.587393 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:08.587498 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:08.587844 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 20:47:08.587902 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:47:08.588932 5236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:08.589733 5545 ts_tablet_manager.cc:1905] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:08.591951 5544 ts_tablet_manager.cc:1905] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:08.596506 5545 ts_tablet_manager.cc:1918] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:47:08.596709 5545 log.cc:1196] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/ce1f3ef2f33f479aabeb467519152920
I20260929 20:47:08.597246 5545 ts_tablet_manager.cc:1939] T ce1f3ef2f33f479aabeb467519152920 P cee9b02b85af478fa1d5fe422edadef1: Deleting consensus metadata
I20260929 20:47:08.599076 5039 catalog_manager.cc:4841] TS cee9b02b85af478fa1d5fe422edadef1 (127.0.97.2:43767): tablet ce1f3ef2f33f479aabeb467519152920 (table test-table [id=99241a569033440e918de643a91cf24a]) successfully deleted
I20260929 20:47:08.599431 5544 ts_tablet_manager.cc:1918] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:47:08.599606 5544 log.cc:1196] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/ce1f3ef2f33f479aabeb467519152920
I20260929 20:47:08.600152 5544 ts_tablet_manager.cc:1939] T ce1f3ef2f33f479aabeb467519152920 P bb9b15507ad94ddc86cf9e086c93ac9f: Deleting consensus metadata
I20260929 20:47:08.601743 5040 catalog_manager.cc:4841] TS bb9b15507ad94ddc86cf9e086c93ac9f (127.0.97.3:39947): tablet ce1f3ef2f33f479aabeb467519152920 (table test-table [id=99241a569033440e918de643a91cf24a]) successfully deleted
I20260929 20:47:08.688862 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 5114
I20260929 20:47:08.702016 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 5244
W20260929 20:47:08.702005 5042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42213 (error 108)
I20260929 20:47:08.703980 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.703448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42213 (local address 127.0.0.1:46110)
0929 20:47:08.703681 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:08.703818 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260929 20:47:08.704279 5042 catalog_manager.cc:4584] TS 739396bb72454fe1870977eb65bb1e5a (127.0.97.1:42213): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce1f3ef2f33f479aabeb467519152920: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42213: connect: Connection refused (error 111)
I20260929 20:47:08.705350 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.703008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36864 (local address 127.0.97.2:43767)
0929 20:47:08.703197 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:08.703202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:08.703244 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:47:08.703301 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:08.703305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:08.703369 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:08.703449 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:08.704806 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:08.704813 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:08.704817 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:08.704832 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:08.704871 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:08.704883 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:08.704978 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:08.705079 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 20:47:08.705140 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:47:08.705957 5366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:08.757679 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.757178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42213 (local address 127.0.0.1:46118)
0929 20:47:08.757390 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:08.757517 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 20:47:08.758749 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.758361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42213 (local address 127.0.0.1:46124)
0929 20:47:08.758509 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:08.758595 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 20:47:08.790499 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 5244
I20260929 20:47:08.804663 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 5374
I20260929 20:47:08.807561 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.805344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50686 (local address 127.0.97.3:39947)
0929 20:47:08.805780 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:08.805785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:08.805807 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:47:08.805854 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:08.805861 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:08.805910 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:08.805990 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:08.806912 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:08.806917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:08.806920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:08.806936 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:08.806968 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:08.806980 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:08.807173 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:08.807253 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 20:47:08.807314 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332}
I20260929 20:47:08.808279 5496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:08.819372 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.818781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42213 (local address 127.0.0.1:46134)
0929 20:47:08.819012 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:08.819168 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 20:47:08.820379 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.820029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42213 (local address 127.0.0.1:46142)
0929 20:47:08.820171 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:08.820247 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 20:47:08.901119 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.900516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42213 (local address 127.0.0.1:46152)
0929 20:47:08.900792 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:08.900907 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 20:47:08.902294 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:08.901822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42213 (local address 127.0.0.1:46160)
0929 20:47:08.902041 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:08.902136 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 20:47:08.922178 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 5374
I20260929 20:47:08.936839 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 5024
I20260929 20:47:08.937908 5085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:09.039654 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 5024
2026-09-29T20:47:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2567 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 20:47:09.075639 388 mini_oidc.cc:152] Starting JWKS server
I20260929 20:47:09.077982 388 webserver.cc:460] Webserver started at https://127.0.0.1:46139/ using document root <none> and password file <none>
I20260929 20:47:09.078236 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 20:47:09.078794 388 webserver.cc:460] Webserver started at http://127.0.0.1:46857/ using document root <none> and password file <none>
2026-09-29T20:47:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:47:09Z Disabled control of system clock
I20260929 20:47:09.100797 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42473
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42473
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46139/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790714786636870-388-0/testchainca.cert with env {}
W20260929 20:47:09.397100 5558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:09.397353 5558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:09.397436 5558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:09.404280 5558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:47:09.404371 5558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:09.404404 5558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:09.404453 5558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:47:09.404495 5558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:47:09.404543 5558 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:47:09.404582 5558 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46139/jwks.json
I20260929 20:47:09.414296 5558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42473
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46139/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790714786636870-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:09.415431 5558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:09.416636 5558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:09.427898 5563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:09.427943 5564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:09.427865 5566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:09.428797 5558 server_base.cc:1034] running on GCE node
I20260929 20:47:09.429477 5558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:09.430703 5558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:09.431875 5558 hybrid_clock.cc:648] HybridClock initialized: now 1790714829431829 us; error 45 us; skew 500 ppm
I20260929 20:47:09.432178 5558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:47:09.434267 5558 webserver.cc:460] Webserver started at http://127.0.97.62:35899/ using document root <none> and password file <none>
I20260929 20:47:09.434888 5558 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:09.434969 5558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:09.435316 5558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:09.437918 5558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "5e0d743bb53e4841b0e5db88d6d989a2"
format_stamp: "Formatted at 2026-09-29 20:47:09 on dist-test-slave-blv7"
server_key: "b0f9a4612bbba2a4764d785c461ff54b"
server_key_iv: "3c383d2395678240bae2a36d4c49386d"
server_key_version: "encryptionkey@0"
I20260929 20:47:09.438624 5558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "5e0d743bb53e4841b0e5db88d6d989a2"
format_stamp: "Formatted at 2026-09-29 20:47:09 on dist-test-slave-blv7"
server_key: "b0f9a4612bbba2a4764d785c461ff54b"
server_key_iv: "3c383d2395678240bae2a36d4c49386d"
server_key_version: "encryptionkey@0"
I20260929 20:47:09.443414 5558 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 20:47:09.446587 5572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:09.447932 5558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:47:09.448104 5558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "5e0d743bb53e4841b0e5db88d6d989a2"
format_stamp: "Formatted at 2026-09-29 20:47:09 on dist-test-slave-blv7"
server_key: "b0f9a4612bbba2a4764d785c461ff54b"
server_key_iv: "3c383d2395678240bae2a36d4c49386d"
server_key_version: "encryptionkey@0"
I20260929 20:47:09.448267 5558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:09.471714 5558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:09.472482 5558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:09.472707 5558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:09.488646 5558 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42473
I20260929 20:47:09.488662 5623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42473 every 8 connection(s)
I20260929 20:47:09.489856 5558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:47:09.493283 5624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:09.495209 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 5558
I20260929 20:47:09.495416 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:47:09.496145 388 external_mini_cluster.cc:1428] Setting key 9ad38e4b0191888e5c6752766c35df61
I20260929 20:47:09.504626 5624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2: Bootstrap starting.
I20260929 20:47:09.507988 5624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:09.509022 5624 log.cc:824] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:09.509604 5627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:09.498045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38900 (local address 127.0.97.62:42473)
0929 20:47:09.499118 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:09.499129 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:09.499171 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:47:09.500832 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:09.500853 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:09.501308 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:09.501648 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:09.502224 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:09.503626 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:09.505346 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:09.506050 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:09.506187 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:09.507361 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:09.507368 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:09.507380 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:09.507874 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:09.507978 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:09.508006 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:09.508129 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:09.508346 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 20:47:09.508610 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":300,"threads_started":1}
I20260929 20:47:09.511795 5624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2: No bootstrap required, opened a new log
I20260929 20:47:09.513023 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:47:09.518993 5624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0d743bb53e4841b0e5db88d6d989a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42473 } }
I20260929 20:47:09.519358 5624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:09.519447 5624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e0d743bb53e4841b0e5db88d6d989a2, State: Initialized, Role: FOLLOWER
I20260929 20:47:09.519982 5624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0d743bb53e4841b0e5db88d6d989a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42473 } }
I20260929 20:47:09.520191 5624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:47:09.520309 5624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:47:09.520447 5624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:09.522392 5624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0d743bb53e4841b0e5db88d6d989a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42473 } }
I20260929 20:47:09.522908 5624 leader_election.cc:302] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e0d743bb53e4841b0e5db88d6d989a2; no voters:
I20260929 20:47:09.523480 5624 leader_election.cc:288] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:47:09.523622 5629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:47:09.524225 5629 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2 [term 1 LEADER]: Becoming Leader. State: Replica: 5e0d743bb53e4841b0e5db88d6d989a2, State: Running, Role: LEADER
I20260929 20:47:09.524667 5629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0d743bb53e4841b0e5db88d6d989a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42473 } }
I20260929 20:47:09.525300 5624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:47:09.527186 5631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e0d743bb53e4841b0e5db88d6d989a2. Latest consensus state: current_term: 1 leader_uuid: "5e0d743bb53e4841b0e5db88d6d989a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0d743bb53e4841b0e5db88d6d989a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42473 } } }
I20260929 20:47:09.527679 5630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e0d743bb53e4841b0e5db88d6d989a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e0d743bb53e4841b0e5db88d6d989a2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42473 } } }
I20260929 20:47:09.528060 5630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2 [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:09.528080 5631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2 [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:09.529469 5637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:47:09.534459 5637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:47:09.542546 5637 catalog_manager.cc:1347] Generated new cluster ID: d92648482f5046dcb7769551e778bf27
I20260929 20:47:09.542652 5637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:47:09.551887 5637 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:47:09.553627 5637 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:47:09.565212 5637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e0d743bb53e4841b0e5db88d6d989a2: Generated new TSK 0
I20260929 20:47:09.566145 5637 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 20:47:09.607709 5654 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 20:47:09.607869 5654 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 20:47:09.608613 5627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:09.580682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38916 (local address 127.0.97.62:42473)
0929 20:47:09.580838 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:09.580842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:09.581193 (+ 351us) server_negotiation.cc:407] Connection header received
0929 20:47:09.581450 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:09.581455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:09.581551 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:09.581706 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0929 20:47:09.582122 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:09.583231 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:09.585212 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:09.585607 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:09.585696 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:09.586255 (+ 559us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 20:47:09.608131 (+ 21876us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 20:47:09.608397 (+ 266us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38916: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 20:47:09.608716 5627 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38916: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 20:47:09.610378 5653 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:09.580542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42473 (local address 127.0.0.1:38916)
0929 20:47:09.581094 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:09.581113 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:09.581253 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:09.581713 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:09.581716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:09.581726 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 20:47:09.581992 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:09.582000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:09.583366 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:09.583372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:09.585057 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:09.585071 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:09.585719 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:09.585763 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:09.585953 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:09.586025 (+ 72us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 20:47:09.608388 (+ 22363us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 20:47:09.608701 (+ 313us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:42473: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":279,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:42473: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46139/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 20:47:09.612658 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 5558
I20260929 20:47:09.613657 5619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:09.702051 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 5558
2026-09-29T20:47:09Z chronyd exiting
I20260929 20:47:09.748307 388 test_util.cc:175] -----------------------------------------------
I20260929 20:47:09.748479 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790714786636870-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (676 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 20:47:09.752230 388 mini_oidc.cc:152] Starting JWKS server
I20260929 20:47:09.754062 388 webserver.cc:460] Webserver started at https://127.0.0.1:37055/ using document root <none> and password file <none>
I20260929 20:47:09.754283 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 20:47:09.754937 388 webserver.cc:460] Webserver started at http://127.0.0.1:46485/ using document root <none> and password file <none>
2026-09-29T20:47:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:47:09Z Disabled control of system clock
I20260929 20:47:09.777281 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34265
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34265
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37055/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790714786636870-388-0/testchainca.cert with env {}
W20260929 20:47:10.078096 5661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:10.078383 5661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:10.078514 5661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:10.085620 5661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:47:10.085728 5661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:10.085794 5661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:10.085876 5661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:47:10.085932 5661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:47:10.086018 5661 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:47:10.086069 5661 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37055/jwks.json
I20260929 20:47:10.095659 5661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34265
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37055/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790714786636870-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:10.096863 5661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:10.098073 5661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:10.108997 5666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:10.108966 5667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:10.109067 5669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:10.109935 5661 server_base.cc:1034] running on GCE node
I20260929 20:47:10.110584 5661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:10.111907 5661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:10.113092 5661 hybrid_clock.cc:648] HybridClock initialized: now 1790714830113049 us; error 58 us; skew 500 ppm
I20260929 20:47:10.113397 5661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:47:10.115410 5661 webserver.cc:460] Webserver started at http://127.0.97.62:37847/ using document root <none> and password file <none>
I20260929 20:47:10.116045 5661 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:10.116139 5661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:10.116420 5661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:10.119088 5661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "6c5345494f5e4e06928a59a09cbbd8cd"
format_stamp: "Formatted at 2026-09-29 20:47:10 on dist-test-slave-blv7"
server_key: "064f1b73787f33c8a33fbc693a97e7d8"
server_key_iv: "4aeb329accdfc3f4b5e624a21bd8efbf"
server_key_version: "encryptionkey@0"
I20260929 20:47:10.119827 5661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "6c5345494f5e4e06928a59a09cbbd8cd"
format_stamp: "Formatted at 2026-09-29 20:47:10 on dist-test-slave-blv7"
server_key: "064f1b73787f33c8a33fbc693a97e7d8"
server_key_iv: "4aeb329accdfc3f4b5e624a21bd8efbf"
server_key_version: "encryptionkey@0"
I20260929 20:47:10.124493 5661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:47:10.127671 5675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:10.128940 5661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:47:10.129096 5661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "6c5345494f5e4e06928a59a09cbbd8cd"
format_stamp: "Formatted at 2026-09-29 20:47:10 on dist-test-slave-blv7"
server_key: "064f1b73787f33c8a33fbc693a97e7d8"
server_key_iv: "4aeb329accdfc3f4b5e624a21bd8efbf"
server_key_version: "encryptionkey@0"
I20260929 20:47:10.129279 5661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:10.153313 5661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:10.154101 5661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:10.154356 5661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:10.170378 5661 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34265
I20260929 20:47:10.170398 5726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34265 every 8 connection(s)
I20260929 20:47:10.171639 5661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:47:10.175056 5727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:10.181442 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 5661
I20260929 20:47:10.181617 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:47:10.182185 388 external_mini_cluster.cc:1428] Setting key 2c653159525519e28915964310bdcdf2
I20260929 20:47:10.184888 5727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd: Bootstrap starting.
I20260929 20:47:10.187815 5727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:10.188819 5727 log.cc:824] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:10.191273 5727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd: No bootstrap required, opened a new log
I20260929 20:47:10.194566 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:10.183944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.0.97.62:34265)
0929 20:47:10.184861 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:10.184872 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:10.184914 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:47:10.186647 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:10.186667 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:10.187139 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:10.187474 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:10.187972 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:10.189309 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:10.190520 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:10.191196 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:10.191363 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:10.192430 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:10.192437 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:10.192450 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:10.192966 (+ 516us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:10.193072 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:10.193095 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:10.193200 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:10.193398 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 20:47:10.193680 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":295,"threads_started":1}
I20260929 20:47:10.197821 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:47:10.198664 5727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c5345494f5e4e06928a59a09cbbd8cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34265 } }
I20260929 20:47:10.198973 5727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:10.199071 5727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c5345494f5e4e06928a59a09cbbd8cd, State: Initialized, Role: FOLLOWER
I20260929 20:47:10.199576 5727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c5345494f5e4e06928a59a09cbbd8cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34265 } }
I20260929 20:47:10.199806 5727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:47:10.199949 5727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:47:10.200103 5727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:10.202353 5727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c5345494f5e4e06928a59a09cbbd8cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34265 } }
I20260929 20:47:10.202754 5727 leader_election.cc:302] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c5345494f5e4e06928a59a09cbbd8cd; no voters:
I20260929 20:47:10.203243 5727 leader_election.cc:288] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:47:10.203353 5732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:47:10.203946 5732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd [term 1 LEADER]: Becoming Leader. State: Replica: 6c5345494f5e4e06928a59a09cbbd8cd, State: Running, Role: LEADER
I20260929 20:47:10.204409 5732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c5345494f5e4e06928a59a09cbbd8cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34265 } }
I20260929 20:47:10.205031 5727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:47:10.206645 5734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c5345494f5e4e06928a59a09cbbd8cd. Latest consensus state: current_term: 1 leader_uuid: "6c5345494f5e4e06928a59a09cbbd8cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c5345494f5e4e06928a59a09cbbd8cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34265 } } }
I20260929 20:47:10.206990 5734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:10.207688 5733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c5345494f5e4e06928a59a09cbbd8cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c5345494f5e4e06928a59a09cbbd8cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34265 } } }
I20260929 20:47:10.208214 5733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:10.208709 5741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:47:10.213977 5741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:47:10.228749 5741 catalog_manager.cc:1347] Generated new cluster ID: 988e92486bb64ea7b5c47867ca324fca
I20260929 20:47:10.228854 5741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:47:10.255342 5741 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:47:10.257134 5741 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:47:10.269649 5741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c5345494f5e4e06928a59a09cbbd8cd: Generated new TSK 0
I20260929 20:47:10.270617 5741 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 20:47:10.306545 5757 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 20:47:10.306644 5757 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 20:47:10.307883 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:10.288162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53092 (local address 127.0.97.62:34265)
0929 20:47:10.288329 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:10.288335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:10.288785 (+ 450us) server_negotiation.cc:407] Connection header received
0929 20:47:10.288962 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:10.288969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:10.289059 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:10.289210 (+ 151us) server_negotiation.cc:226] Negotiated authn=JWT
0929 20:47:10.289588 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:10.290673 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:10.292005 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:10.292647 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:10.292777 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:10.293178 (+ 401us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 20:47:10.307245 (+ 14067us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 20:47:10.307595 (+ 350us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53092: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 20:47:10.307945 5756 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:10.288136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34265 (local address 127.0.0.1:53092)
0929 20:47:10.288693 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:10.288708 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:10.288823 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:10.289214 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:10.289217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:10.289226 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 20:47:10.289450 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:10.289457 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:10.290780 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:10.290783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:10.291878 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:10.291889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:10.292749 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:10.292800 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:10.292922 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:10.292950 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 20:47:10.307451 (+ 14501us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 20:47:10.307661 (+ 210us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:34265: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":296,"threads_started":1}
W20260929 20:47:10.307996 5730 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53092: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37055/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260929 20:47:10.309615 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 5661
I20260929 20:47:10.310660 5722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:10.399765 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 5661
2026-09-29T20:47:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (700 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 20:47:10.452759 388 mini_oidc.cc:152] Starting JWKS server
I20260929 20:47:10.454595 388 webserver.cc:460] Webserver started at https://127.0.0.1:46607/ using document root <none> and password file <none>
I20260929 20:47:10.454802 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 20:47:10.455449 388 webserver.cc:460] Webserver started at http://127.0.0.1:35925/ using document root <none> and password file <none>
2026-09-29T20:47:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:47:10Z Disabled control of system clock
I20260929 20:47:10.478864 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35449
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35449
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46607/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 20:47:10.773005 5764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:10.773304 5764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:10.773432 5764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:10.780267 5764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:47:10.780368 5764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:10.780431 5764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:10.780503 5764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:47:10.780562 5764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:47:10.780642 5764 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:47:10.780690 5764 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46607/jwks.json
I20260929 20:47:10.789287 5764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46607/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:10.790436 5764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:10.792186 5764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:10.802671 5769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:10.802728 5770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:10.803360 5764 server_base.cc:1034] running on GCE node
W20260929 20:47:10.803871 5772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:10.804773 5764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:10.806150 5764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:10.807328 5764 hybrid_clock.cc:648] HybridClock initialized: now 1790714830807277 us; error 47 us; skew 500 ppm
I20260929 20:47:10.807626 5764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:47:10.809724 5764 webserver.cc:460] Webserver started at http://127.0.97.62:41567/ using document root <none> and password file <none>
I20260929 20:47:10.810331 5764 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:10.810417 5764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:10.810707 5764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:10.813398 5764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "07599f4e93ad4b03bbb7da1b85bf7437"
format_stamp: "Formatted at 2026-09-29 20:47:10 on dist-test-slave-blv7"
server_key: "d64d61db04a6c77fa3be9839b7ed8120"
server_key_iv: "0d269992238a447d8d46d281cc01a76a"
server_key_version: "encryptionkey@0"
I20260929 20:47:10.814110 5764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "07599f4e93ad4b03bbb7da1b85bf7437"
format_stamp: "Formatted at 2026-09-29 20:47:10 on dist-test-slave-blv7"
server_key: "d64d61db04a6c77fa3be9839b7ed8120"
server_key_iv: "0d269992238a447d8d46d281cc01a76a"
server_key_version: "encryptionkey@0"
I20260929 20:47:10.818719 5764 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260929 20:47:10.821815 5778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:10.823113 5764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:47:10.823253 5764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "07599f4e93ad4b03bbb7da1b85bf7437"
format_stamp: "Formatted at 2026-09-29 20:47:10 on dist-test-slave-blv7"
server_key: "d64d61db04a6c77fa3be9839b7ed8120"
server_key_iv: "0d269992238a447d8d46d281cc01a76a"
server_key_version: "encryptionkey@0"
I20260929 20:47:10.823427 5764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:10.841873 5764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:10.842623 5764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:10.842860 5764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:10.859193 5764 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35449
I20260929 20:47:10.859213 5829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35449 every 8 connection(s)
I20260929 20:47:10.860379 5764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:47:10.863309 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 5764
I20260929 20:47:10.863507 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:47:10.864115 5830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:10.864233 388 external_mini_cluster.cc:1428] Setting key fc674bf12e8ced558994b2139dc7ab0a
I20260929 20:47:10.876703 5830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437: Bootstrap starting.
I20260929 20:47:10.876747 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:10.865974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35514 (local address 127.0.97.62:35449)
0929 20:47:10.866881 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:10.866891 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:10.866930 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:47:10.868539 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:10.868559 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:10.868980 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:10.869293 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:10.869801 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:10.871127 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:10.872798 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:10.873420 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:10.873549 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:10.874662 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:10.874669 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:10.874681 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:10.875242 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:10.875310 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:10.875329 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:10.875410 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:10.875604 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 20:47:10.875871 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":323,"threads_started":1}
I20260929 20:47:10.879604 5830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:10.880000 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:47:10.880721 5830 log.cc:824] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:10.883356 5830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437: No bootstrap required, opened a new log
I20260929 20:47:10.889802 5830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07599f4e93ad4b03bbb7da1b85bf7437" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35449 } }
I20260929 20:47:10.890173 5830 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:10.890262 5830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07599f4e93ad4b03bbb7da1b85bf7437, State: Initialized, Role: FOLLOWER
I20260929 20:47:10.890852 5830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07599f4e93ad4b03bbb7da1b85bf7437" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35449 } }
I20260929 20:47:10.891141 5830 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:47:10.891283 5830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:47:10.891444 5830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:10.893347 5830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07599f4e93ad4b03bbb7da1b85bf7437" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35449 } }
I20260929 20:47:10.893745 5830 leader_election.cc:302] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07599f4e93ad4b03bbb7da1b85bf7437; no voters:
I20260929 20:47:10.894191 5830 leader_election.cc:288] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:47:10.894315 5835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:47:10.894956 5835 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437 [term 1 LEADER]: Becoming Leader. State: Replica: 07599f4e93ad4b03bbb7da1b85bf7437, State: Running, Role: LEADER
I20260929 20:47:10.895424 5835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07599f4e93ad4b03bbb7da1b85bf7437" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35449 } }
I20260929 20:47:10.896191 5830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:47:10.897976 5837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07599f4e93ad4b03bbb7da1b85bf7437. Latest consensus state: current_term: 1 leader_uuid: "07599f4e93ad4b03bbb7da1b85bf7437" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07599f4e93ad4b03bbb7da1b85bf7437" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35449 } } }
I20260929 20:47:10.898324 5837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437 [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:10.899595 5836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07599f4e93ad4b03bbb7da1b85bf7437" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07599f4e93ad4b03bbb7da1b85bf7437" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35449 } } }
I20260929 20:47:10.899902 5836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437 [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:10.900364 5844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:47:10.905169 5844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:47:10.912653 5844 catalog_manager.cc:1347] Generated new cluster ID: 2fa62a3706034e50acd745be7846ddfa
I20260929 20:47:10.912746 5844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:47:10.945941 5844 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:47:10.947232 5844 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:47:10.958513 5844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07599f4e93ad4b03bbb7da1b85bf7437: Generated new TSK 0
I20260929 20:47:10.959295 5844 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:47:10.988417 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:10.980548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35528 (local address 127.0.97.62:35449)
0929 20:47:10.980730 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:10.980736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:10.981060 (+ 324us) server_negotiation.cc:407] Connection header received
0929 20:47:10.981248 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:10.981254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:10.981338 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:10.981509 (+ 171us) server_negotiation.cc:226] Negotiated authn=JWT
0929 20:47:10.981906 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:10.983056 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:10.984331 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:10.985036 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:10.985153 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:10.985570 (+ 417us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 20:47:10.987798 (+ 2228us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46607/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 20:47:10.988095 (+ 297us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35528: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46607/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 20:47:10.988566 5833 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35528: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46607/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 20:47:10.990060 5859 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:10.980396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35449 (local address 127.0.0.1:35528)
0929 20:47:10.980970 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:10.980985 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:10.981093 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:10.981518 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:10.981521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:10.981530 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 20:47:10.981777 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:10.981785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:10.983171 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:10.983175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:10.984206 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:10.984217 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:10.985133 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:10.985188 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:10.985308 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:10.985337 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 20:47:10.987985 (+ 2648us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46607/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 20:47:10.989673 (+ 1688us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:35449: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46607/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":1421,"thread_start_us":323,"threads_started":1}
I20260929 20:47:10.991326 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 5764
I20260929 20:47:10.992360 5825 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 20:47:10.999799 5860 webserver.cc:533] Webserver: sslize failed:
E20260929 20:47:10.999881 5860 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 20:47:11.081187 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 5764
2026-09-29T20:47:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (679 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T20:47:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:47:11Z Disabled control of system clock
I20260929 20:47:11.152928 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45367
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45367
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790714786636870-388-0/testchainca.cert with env {}
W20260929 20:47:11.448102 5865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:11.448374 5865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:11.448495 5865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:11.455468 5865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:47:11.455564 5865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:11.455627 5865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:11.455706 5865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:47:11.455765 5865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:47:11.455847 5865 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:47:11.455909 5865 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 20:47:11.465476 5865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45367
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790714786636870-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:11.466770 5865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:11.468021 5865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:11.478777 5870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:11.478823 5871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:11.478852 5873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:11.479420 5865 server_base.cc:1034] running on GCE node
I20260929 20:47:11.480113 5865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:11.481400 5865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:11.482583 5865 hybrid_clock.cc:648] HybridClock initialized: now 1790714831482540 us; error 34 us; skew 500 ppm
I20260929 20:47:11.482879 5865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:47:11.488796 5865 webserver.cc:460] Webserver started at http://127.0.97.62:35583/ using document root <none> and password file <none>
I20260929 20:47:11.489421 5865 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:11.489516 5865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:11.489813 5865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:11.492516 5865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "16d5340681dd4016b535645307082972"
format_stamp: "Formatted at 2026-09-29 20:47:11 on dist-test-slave-blv7"
server_key: "62b5128c176c287ba888d530eb0be913"
server_key_iv: "246a99c2980ea640aceba57be42a13a7"
server_key_version: "encryptionkey@0"
I20260929 20:47:11.493235 5865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "16d5340681dd4016b535645307082972"
format_stamp: "Formatted at 2026-09-29 20:47:11 on dist-test-slave-blv7"
server_key: "62b5128c176c287ba888d530eb0be913"
server_key_iv: "246a99c2980ea640aceba57be42a13a7"
server_key_version: "encryptionkey@0"
I20260929 20:47:11.497865 5865 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 20:47:11.500926 5879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:11.502139 5865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 20:47:11.502295 5865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "16d5340681dd4016b535645307082972"
format_stamp: "Formatted at 2026-09-29 20:47:11 on dist-test-slave-blv7"
server_key: "62b5128c176c287ba888d530eb0be913"
server_key_iv: "246a99c2980ea640aceba57be42a13a7"
server_key_version: "encryptionkey@0"
I20260929 20:47:11.502475 5865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:11.514238 5865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:11.514941 5865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:11.515208 5865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:11.531668 5865 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45367
I20260929 20:47:11.531687 5930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45367 every 8 connection(s)
I20260929 20:47:11.532877 5865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:47:11.536248 5931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:11.537585 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 5865
I20260929 20:47:11.537760 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:47:11.538519 388 external_mini_cluster.cc:1428] Setting key 489f38a63d46025182a2ff1ac121c339
I20260929 20:47:11.546566 5931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972: Bootstrap starting.
I20260929 20:47:11.549412 5931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:11.550417 5931 log.cc:824] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:11.550649 5934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:11.540370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51216 (local address 127.0.97.62:45367)
0929 20:47:11.541292 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:11.541302 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:11.541345 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:47:11.542885 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:11.542904 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:11.543360 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:11.543698 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:11.544203 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:11.545449 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:11.546677 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:11.547346 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:11.547507 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:11.548560 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:11.548567 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:11.548579 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:11.549064 (+ 485us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:11.549148 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:11.549167 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:11.549265 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:11.549479 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 20:47:11.549753 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":344,"threads_started":1}
I20260929 20:47:11.552907 5931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972: No bootstrap required, opened a new log
I20260929 20:47:11.553921 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:47:11.559998 5931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d5340681dd4016b535645307082972" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } }
I20260929 20:47:11.560309 5931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:11.560389 5931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d5340681dd4016b535645307082972, State: Initialized, Role: FOLLOWER
I20260929 20:47:11.560868 5931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d5340681dd4016b535645307082972" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } }
I20260929 20:47:11.561076 5931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:47:11.561185 5931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:47:11.561321 5931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:11.563242 5931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d5340681dd4016b535645307082972" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } }
I20260929 20:47:11.563671 5931 leader_election.cc:302] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16d5340681dd4016b535645307082972; no voters:
I20260929 20:47:11.564178 5931 leader_election.cc:288] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:47:11.564321 5936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:47:11.565927 5931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:47:11.566236 5936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972 [term 1 LEADER]: Becoming Leader. State: Replica: 16d5340681dd4016b535645307082972, State: Running, Role: LEADER
I20260929 20:47:11.566665 5936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d5340681dd4016b535645307082972" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } }
I20260929 20:47:11.569372 5937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16d5340681dd4016b535645307082972" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d5340681dd4016b535645307082972" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } } }
I20260929 20:47:11.569743 5937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972 [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:11.570106 5938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16d5340681dd4016b535645307082972. Latest consensus state: current_term: 1 leader_uuid: "16d5340681dd4016b535645307082972" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d5340681dd4016b535645307082972" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45367 } } }
I20260929 20:47:11.570369 5938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972 [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:11.571823 5947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:47:11.575956 5947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:47:11.583184 5947 catalog_manager.cc:1347] Generated new cluster ID: d657e4fb1bf3462d8ff8308ff2904f23
I20260929 20:47:11.583276 5947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:47:11.609547 5947 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:47:11.611367 5947 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:47:11.625988 5947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16d5340681dd4016b535645307082972: Generated new TSK 0
I20260929 20:47:11.627064 5947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:47:12.192179 5934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:11.646155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51218 (local address 127.0.97.62:45367)
0929 20:47:11.646335 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:11.646341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:11.646647 (+ 306us) server_negotiation.cc:407] Connection header received
0929 20:47:11.646856 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:11.646860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:11.646923 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:11.647057 (+ 134us) server_negotiation.cc:226] Negotiated authn=JWT
0929 20:47:11.647516 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:11.648228 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:11.650128 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:11.650516 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:11.650600 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:11.651132 (+ 532us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 20:47:12.191644 (+540512us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 20:47:12.191947 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51218: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 20:47:12.192276 5934 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51218: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 20:47:12.192566 5960 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:11.645992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45367 (local address 127.0.0.1:51218)
0929 20:47:11.646552 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:11.646570 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:11.646711 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:11.647085 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:11.647090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:11.647103 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 20:47:11.647378 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:11.647399 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:11.648357 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:11.648363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:11.649988 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:11.650003 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:11.650616 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:11.650664 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:11.650854 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:11.650898 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 20:47:12.191935 (+541037us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 20:47:12.192203 (+ 268us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:45367: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":286,"threads_started":1}
I20260929 20:47:12.193938 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 5865
I20260929 20:47:12.194909 5926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:12.290728 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 5865
2026-09-29T20:47:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1190 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (376 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1364 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:47:14 dist-test-slave-blv7 krb5kdc[5976](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:47:14 dist-test-slave-blv7 krb5kdc[5976](info): set up 2 sockets
Sep 29 20:47:14 dist-test-slave-blv7 krb5kdc[5976](info): commencing operation
krb5kdc: starting...
W20260929 20:47:15.019418 388 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:47:15 dist-test-slave-blv7 krb5kdc[5976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:47:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:47:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:47:15.195629 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35225
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35225
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:15.492563 5989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:15.492867 5989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:15.492991 5989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:15.500017 5989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:47:15.500124 5989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:15.500200 5989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:15.500275 5989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:47:15.500334 5989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:47:15.509716 5989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:15.510897 5989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:15.512132 5989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:15.523062 5994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:15.523033 5997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:15.523109 5995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:15.523600 5989 server_base.cc:1034] running on GCE node
I20260929 20:47:15.524304 5989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:15.525539 5989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:15.526722 5989 hybrid_clock.cc:648] HybridClock initialized: now 1790714835526675 us; error 34 us; skew 500 ppm
I20260929 20:47:15.527040 5989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:15 dist-test-slave-blv7 krb5kdc[5976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:15.529716 5989 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:47:15.531042 5989 webserver.cc:460] Webserver started at http://127.0.97.62:34463/ using document root <none> and password file <none>
I20260929 20:47:15.531664 5989 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:15.531775 5989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:15.532045 5989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:15.534741 5989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "24ba7d6275f64ddb9e9394dea956579f"
format_stamp: "Formatted at 2026-09-29 20:47:15 on dist-test-slave-blv7"
server_key: "2c0f9520f6e4d0c9dbcbb9115015584b"
server_key_iv: "94a69879b428e79ef6e63d2f3c080d72"
server_key_version: "encryptionkey@0"
I20260929 20:47:15.535480 5989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "24ba7d6275f64ddb9e9394dea956579f"
format_stamp: "Formatted at 2026-09-29 20:47:15 on dist-test-slave-blv7"
server_key: "2c0f9520f6e4d0c9dbcbb9115015584b"
server_key_iv: "94a69879b428e79ef6e63d2f3c080d72"
server_key_version: "encryptionkey@0"
I20260929 20:47:15.540218 5989 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 20:47:15.543380 6004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:15.544581 5989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:47:15.544724 5989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "24ba7d6275f64ddb9e9394dea956579f"
format_stamp: "Formatted at 2026-09-29 20:47:15 on dist-test-slave-blv7"
server_key: "2c0f9520f6e4d0c9dbcbb9115015584b"
server_key_iv: "94a69879b428e79ef6e63d2f3c080d72"
server_key_version: "encryptionkey@0"
I20260929 20:47:15.544883 5989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:15.568536 5989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:15.572198 5989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:15.572438 5989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:15.588644 5989 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35225
I20260929 20:47:15.588685 6055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35225 every 8 connection(s)
I20260929 20:47:15.589849 5989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:47:15.590610 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 5989
I20260929 20:47:15.590802 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:47:15.591472 388 external_mini_cluster.cc:1428] Setting key 0625bf0adccefae3f1e1933b7a3f7261
I20260929 20:47:15.593608 6056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:15.605185 6056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f: Bootstrap starting.
I20260929 20:47:15.608655 6056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:15.609835 6056 log.cc:824] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:47:15 dist-test-slave-blv7 krb5kdc[5976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:47:15.612737 6056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f: No bootstrap required, opened a new log
I20260929 20:47:15.619444 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:15.593364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33460 (local address 127.0.97.62:35225)
0929 20:47:15.599901 (+ 6537us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:15.599912 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:15.599952 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:47:15.601451 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:15.601471 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:15.601858 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:15.602166 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:15.603744 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:15.605001 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:15.606660 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:15.607336 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:15.607461 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:15.611764 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:15.611774 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:15.611791 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:15.611830 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:15.615383 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:15.615964 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:15.615972 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:15.615980 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:15.616180 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:15.616476 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:15.616480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:15.616483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:15.617126 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:15.617383 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:15.617717 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 20:47:15.618143 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6337,"thread_start_us":6128,"threads_started":1}
I20260929 20:47:15.621888 6056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ba7d6275f64ddb9e9394dea956579f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35225 } }
I20260929 20:47:15.622184 6056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:15.622249 6056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ba7d6275f64ddb9e9394dea956579f, State: Initialized, Role: FOLLOWER
I20260929 20:47:15.622710 6056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ba7d6275f64ddb9e9394dea956579f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35225 } }
I20260929 20:47:15.622918 6056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:47:15.623005 6056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:47:15.623147 6056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:15.625041 6056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ba7d6275f64ddb9e9394dea956579f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35225 } }
I20260929 20:47:15.625435 6056 leader_election.cc:302] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24ba7d6275f64ddb9e9394dea956579f; no voters:
I20260929 20:47:15.625959 6056 leader_election.cc:288] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:47:15.626106 6061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:47:15.626716 6061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f [term 1 LEADER]: Becoming Leader. State: Replica: 24ba7d6275f64ddb9e9394dea956579f, State: Running, Role: LEADER
I20260929 20:47:15.627259 6061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ba7d6275f64ddb9e9394dea956579f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35225 } }
I20260929 20:47:15.627794 6056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:47:15.629848 6063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24ba7d6275f64ddb9e9394dea956579f. Latest consensus state: current_term: 1 leader_uuid: "24ba7d6275f64ddb9e9394dea956579f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ba7d6275f64ddb9e9394dea956579f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35225 } } }
I20260929 20:47:15.630189 6063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:15.629992 6062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24ba7d6275f64ddb9e9394dea956579f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ba7d6275f64ddb9e9394dea956579f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35225 } } }
I20260929 20:47:15.630330 6062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:15.632848 6070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:47:15.637169 6070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:47:15.644846 6070 catalog_manager.cc:1347] Generated new cluster ID: f6ff5d6567c74d42999903c9c38af63c
I20260929 20:47:15.644965 6070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:47:15.658993 6070 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:47:15.660763 6070 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:47:15.671854 6070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24ba7d6275f64ddb9e9394dea956579f: Generated new TSK 0
I20260929 20:47:15.672835 6070 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:47:15.714535 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35225
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:16.019125 6082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:16.019403 6082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:16.019541 6082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:16.026623 6082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:16.026733 6082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:16.026887 6082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:47:16.036418 6082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:16.037580 6082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:16.038774 6082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:16.050948 6090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:16.050895 6088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:16.051108 6087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:16.051440 6082 server_base.cc:1034] running on GCE node
I20260929 20:47:16.052131 6082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:16.053025 6082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:16.054235 6082 hybrid_clock.cc:648] HybridClock initialized: now 1790714836054184 us; error 42 us; skew 500 ppm
I20260929 20:47:16.054548 6082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:16 dist-test-slave-blv7 krb5kdc[5976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:16.057652 6082 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:47:16.059113 6082 webserver.cc:460] Webserver started at http://127.0.97.1:35567/ using document root <none> and password file <none>
I20260929 20:47:16.059799 6082 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:16.059922 6082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:16.060207 6082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:16.063248 6082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "98e1f6171409474db711cebae011c597"
format_stamp: "Formatted at 2026-09-29 20:47:16 on dist-test-slave-blv7"
server_key: "478421bd9a787f8fb9c355b9bcbd6157"
server_key_iv: "40a5a86e3e1d814745fb79cf1ba65858"
server_key_version: "encryptionkey@0"
I20260929 20:47:16.064045 6082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "98e1f6171409474db711cebae011c597"
format_stamp: "Formatted at 2026-09-29 20:47:16 on dist-test-slave-blv7"
server_key: "478421bd9a787f8fb9c355b9bcbd6157"
server_key_iv: "40a5a86e3e1d814745fb79cf1ba65858"
server_key_version: "encryptionkey@0"
I20260929 20:47:16.069209 6082 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:47:16.072408 6097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:16.073678 6082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:47:16.073820 6082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "98e1f6171409474db711cebae011c597"
format_stamp: "Formatted at 2026-09-29 20:47:16 on dist-test-slave-blv7"
server_key: "478421bd9a787f8fb9c355b9bcbd6157"
server_key_iv: "40a5a86e3e1d814745fb79cf1ba65858"
server_key_version: "encryptionkey@0"
I20260929 20:47:16.074002 6082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:16.085824 6082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:16.089661 6082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:16.089907 6082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:16.090773 6082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:16.092027 6082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:16.092100 6082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:16.092218 6082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:16.092269 6082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:16.120908 6082 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36553
I20260929 20:47:16.120950 6209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36553 every 8 connection(s)
I20260929 20:47:16.122066 6082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:47:16.123245 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 6082
I20260929 20:47:16.123430 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:47:16.124177 388 external_mini_cluster.cc:1428] Setting key 6dae0b97b05255a593e97f9396974b7d
Sep 29 20:47:16 dist-test-slave-blv7 krb5kdc[5976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:47:16.141662 6212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:16.124635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40443 (local address 127.0.97.62:35225)
0929 20:47:16.125061 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:16.125065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:16.126136 (+ 1071us) server_negotiation.cc:407] Connection header received
0929 20:47:16.127920 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:16.127925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:16.128003 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:16.128100 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:16.129654 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:16.130397 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:16.132006 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:16.132419 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:16.132524 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:16.136096 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:16.136100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:16.136104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:16.136134 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:16.138644 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:16.139489 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:16.139493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:16.139495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:16.139608 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:16.140040 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:16.140042 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:16.140044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:16.140304 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:16.140404 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:16.141241 (+ 837us) server_negotiation.cc:299] Negotiation successful
0929 20:47:16.141412 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":209,"threads_started":1}
I20260929 20:47:16.142652 6213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:16.125199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35225 (local address 127.0.97.1:40443)
0929 20:47:16.125979 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:16.126028 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:16.127661 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:16.128345 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:16.128354 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:16.128837 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:16.129498 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:16.129516 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:16.130537 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:16.130541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:16.131884 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:16.131899 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:16.132653 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:16.132660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:16.132882 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:16.133501 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:16.133521 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:16.135881 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:16.138786 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:16.138795 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:16.138812 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:16.139386 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:16.139701 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:16.139706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:16.139709 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:16.139934 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:16.140422 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:16.140430 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:16.140881 (+ 451us) client_negotiation.cc:769] Sending connection context
0929 20:47:16.141166 (+ 285us) client_negotiation.cc:240] Negotiation successful
0929 20:47:16.141647 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":302,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 20:47:16.144127 6210 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35225
I20260929 20:47:16.144403 6210 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:16.145077 6210 heartbeater.cc:511] Master 127.0.97.62:35225 requested a full tablet report, sending...
I20260929 20:47:16.147315 6021 ts_manager.cc:194] Registered new tserver with Master: 98e1f6171409474db711cebae011c597 (127.0.97.1:36553)
I20260929 20:47:16.149088 6021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:47:16.166698 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35225
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:16.474128 6216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:16.474399 6216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:16.474539 6216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:16.481604 6216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:16.481714 6216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:16.481858 6216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:47:16.491088 6216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:16.492290 6216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:16.493522 6216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:16.505440 6221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:16.505544 6222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:16.505523 6224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:16.506465 6216 server_base.cc:1034] running on GCE node
I20260929 20:47:16.507081 6216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:16.507944 6216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:16.509132 6216 hybrid_clock.cc:648] HybridClock initialized: now 1790714836509082 us; error 46 us; skew 500 ppm
I20260929 20:47:16.509423 6216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:16 dist-test-slave-blv7 krb5kdc[5976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:16.515311 6216 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:47:16.516716 6216 webserver.cc:460] Webserver started at http://127.0.97.2:44021/ using document root <none> and password file <none>
I20260929 20:47:16.517352 6216 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:16.517443 6216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:16.517673 6216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:16.520382 6216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "d5fc904fce394725bc9067be7652db7c"
format_stamp: "Formatted at 2026-09-29 20:47:16 on dist-test-slave-blv7"
server_key: "d08f45b822dee3f8ee7e1d9bc5bd3db7"
server_key_iv: "ac70707bafdcd5058675fec08e2a4831"
server_key_version: "encryptionkey@0"
I20260929 20:47:16.521076 6216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d5fc904fce394725bc9067be7652db7c"
format_stamp: "Formatted at 2026-09-29 20:47:16 on dist-test-slave-blv7"
server_key: "d08f45b822dee3f8ee7e1d9bc5bd3db7"
server_key_iv: "ac70707bafdcd5058675fec08e2a4831"
server_key_version: "encryptionkey@0"
I20260929 20:47:16.525696 6216 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 20:47:16.528767 6231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:16.530078 6216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:47:16.530215 6216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "d5fc904fce394725bc9067be7652db7c"
format_stamp: "Formatted at 2026-09-29 20:47:16 on dist-test-slave-blv7"
server_key: "d08f45b822dee3f8ee7e1d9bc5bd3db7"
server_key_iv: "ac70707bafdcd5058675fec08e2a4831"
server_key_version: "encryptionkey@0"
I20260929 20:47:16.530382 6216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:16.545548 6216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:16.549134 6216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:16.549365 6216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:16.550217 6216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:16.551422 6216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:16.551487 6216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:16.551594 6216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:16.551649 6216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:16.579732 6216 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33389
I20260929 20:47:16.579777 6343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33389 every 8 connection(s)
I20260929 20:47:16.580929 6216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:47:16.583415 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 6216
I20260929 20:47:16.583551 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:47:16.583933 388 external_mini_cluster.cc:1428] Setting key faa56f9208f4c9d2c45437b1ef97179d
Sep 29 20:47:16 dist-test-slave-blv7 krb5kdc[5976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:47:16.605316 6212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:16.583244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32873 (local address 127.0.97.62:35225)
0929 20:47:16.583436 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:16.583441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:16.587743 (+ 4302us) server_negotiation.cc:407] Connection header received
0929 20:47:16.589496 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:16.589501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:16.589603 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:16.589703 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:16.591121 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:16.591915 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:16.594119 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:16.594562 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:16.594653 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:16.598615 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:16.598622 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:16.598627 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:16.598661 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:16.601863 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:16.602736 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:16.602742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:16.602746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:16.602900 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:16.603407 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:16.603412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:16.603415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:16.603757 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:16.603886 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:16.604740 (+ 854us) server_negotiation.cc:299] Negotiation successful
0929 20:47:16.604977 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:47:16.606129 6346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:16.583768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35225 (local address 127.0.97.2:32873)
0929 20:47:16.587581 (+ 3813us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:16.587625 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:16.589250 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:16.589941 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:16.589949 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:16.590430 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:16.590952 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:16.590968 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:16.592101 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:16.592106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:16.593983 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:16.593997 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:16.594749 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:16.594755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:16.595054 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:16.595735 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:16.595763 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:16.598407 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:16.602008 (+ 3601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:16.602017 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:16.602034 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:16.602604 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:16.603007 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:16.603012 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:16.603038 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:16.603286 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:16.603888 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:16.603896 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:16.604320 (+ 424us) client_negotiation.cc:769] Sending connection context
0929 20:47:16.604641 (+ 321us) client_negotiation.cc:240] Negotiation successful
0929 20:47:16.605072 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3535,"thread_start_us":3354,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 20:47:16.607496 6344 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35225
I20260929 20:47:16.607780 6344 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:16.608449 6344 heartbeater.cc:511] Master 127.0.97.62:35225 requested a full tablet report, sending...
I20260929 20:47:16.609757 6021 ts_manager.cc:194] Registered new tserver with Master: d5fc904fce394725bc9067be7652db7c (127.0.97.2:33389)
I20260929 20:47:16.610522 6021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:32873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:47:16.628371 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35225
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:16.930445 6349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:16.930733 6349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:16.930874 6349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:16.937938 6349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:16.938030 6349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:16.938161 6349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:47:16.947573 6349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:16.948815 6349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:16.950039 6349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:16.962312 6354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:16.962276 6357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:16.962344 6355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:16.963243 6349 server_base.cc:1034] running on GCE node
I20260929 20:47:16.963814 6349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:16.964697 6349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:16.965924 6349 hybrid_clock.cc:648] HybridClock initialized: now 1790714836965880 us; error 38 us; skew 500 ppm
I20260929 20:47:16.966190 6349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:16 dist-test-slave-blv7 krb5kdc[5976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:16.969044 6349 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:47:16.970359 6349 webserver.cc:460] Webserver started at http://127.0.97.3:42633/ using document root <none> and password file <none>
I20260929 20:47:16.970960 6349 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:16.971095 6349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:16.971342 6349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:16.974018 6349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "dfb63231d8ce4af4b6c84fc672fa6b24"
format_stamp: "Formatted at 2026-09-29 20:47:16 on dist-test-slave-blv7"
server_key: "a17e675e89bedc9a03fdfe9bf3961960"
server_key_iv: "4480daecb1b010f24cf5ac1f59c44396"
server_key_version: "encryptionkey@0"
I20260929 20:47:16.974700 6349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "dfb63231d8ce4af4b6c84fc672fa6b24"
format_stamp: "Formatted at 2026-09-29 20:47:16 on dist-test-slave-blv7"
server_key: "a17e675e89bedc9a03fdfe9bf3961960"
server_key_iv: "4480daecb1b010f24cf5ac1f59c44396"
server_key_version: "encryptionkey@0"
I20260929 20:47:16.979566 6349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:47:16.982649 6364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:16.983880 6349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 20:47:16.984004 6349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "dfb63231d8ce4af4b6c84fc672fa6b24"
format_stamp: "Formatted at 2026-09-29 20:47:16 on dist-test-slave-blv7"
server_key: "a17e675e89bedc9a03fdfe9bf3961960"
server_key_iv: "4480daecb1b010f24cf5ac1f59c44396"
server_key_version: "encryptionkey@0"
I20260929 20:47:16.984159 6349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:17.004034 6349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:17.007927 6349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:17.008152 6349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:17.009060 6349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:17.010265 6349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:17.010332 6349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:17.010421 6349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:17.010461 6349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:17.038936 6349 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34687
I20260929 20:47:17.038990 6476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34687 every 8 connection(s)
I20260929 20:47:17.040424 6349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:47:17.044673 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 6349
I20260929 20:47:17.044821 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:47:17.045233 388 external_mini_cluster.cc:1428] Setting key 8b544d74a394f6b029d7d4b1d9bc334a
Sep 29 20:47:17 dist-test-slave-blv7 krb5kdc[5976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:47:17.058931 6212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:17.042784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49609 (local address 127.0.97.62:35225)
0929 20:47:17.042968 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:17.042974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:17.044289 (+ 1315us) server_negotiation.cc:407] Connection header received
0929 20:47:17.046295 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:17.046300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:17.046381 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:17.046463 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:17.048018 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:17.048768 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:17.050299 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:17.050702 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:17.050805 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:17.053963 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:17.053968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:17.053971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:17.054002 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:17.056213 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:17.056848 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:17.056851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:17.056853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:17.056965 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:17.057342 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:17.057346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:17.057348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:17.057597 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:17.057717 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:17.058383 (+ 666us) server_negotiation.cc:299] Negotiation successful
0929 20:47:17.058604 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:47:17.059983 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:17.043349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35225 (local address 127.0.97.3:49609)
0929 20:47:17.044121 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:17.044170 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:17.046022 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:17.046712 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:17.046725 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:17.047349 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:17.047879 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:17.047896 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:17.048910 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:17.048914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:17.050175 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:17.050187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:17.050902 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:17.050909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:17.051176 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:17.051742 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:17.051768 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:17.053768 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:17.056332 (+ 2564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:17.056340 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:17.056356 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:17.056736 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:17.057057 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:17.057070 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:17.057074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:17.057236 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:17.057707 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:17.057714 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:17.058027 (+ 313us) client_negotiation.cc:769] Sending connection context
0929 20:47:17.058262 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 20:47:17.058776 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":280,"threads_started":1}
I20260929 20:47:17.061170 6477 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35225
I20260929 20:47:17.061424 6477 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:17.062072 6477 heartbeater.cc:511] Master 127.0.97.62:35225 requested a full tablet report, sending...
I20260929 20:47:17.063413 6021 ts_manager.cc:194] Registered new tserver with Master: dfb63231d8ce4af4b6c84fc672fa6b24 (127.0.97.3:34687)
I20260929 20:47:17.064198 6021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49609
I20260929 20:47:17.072043 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/test-admin.keytab.
Sep 29 20:47:17 dist-test-slave-blv7 krb5kdc[5976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:17.096097 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 20:47:17.100651 6484 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:17.098475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52838 (local address 127.0.97.1:36553)
0929 20:47:17.099005 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:17.099031 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:17.099064 (+ 33us) server_negotiation.cc:407] Connection header received
0929 20:47:17.099167 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:17.099175 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:17.099388 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:17.099533 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:17.100329 (+ 796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52838: BlockingRecv error: recv got EOF from 127.0.0.1:52838 (error 108)
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":253,"threads_started":1}
W20260929 20:47:17.100817 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/krb5cc' of type FILE
Sep 29 20:47:17 dist-test-slave-blv7 krb5kdc[5976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:17.102272 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 20:47:17 dist-test-slave-blv7 krb5kdc[5976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:17.103509 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 20:47:17 dist-test-slave-blv7 krb5kdc[5976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:47:17.117727 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:17.104873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52848 (local address 127.0.97.1:36553)
0929 20:47:17.105036 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:17.105042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:17.105379 (+ 337us) server_negotiation.cc:407] Connection header received
0929 20:47:17.105457 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:17.105461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:17.105524 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:17.105613 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:17.106519 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:17.107605 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:17.108836 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:17.109377 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:17.109481 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:17.112803 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:17.112814 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:17.112828 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:17.112859 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:17.115341 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:17.115918 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:17.115926 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:17.115929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:17.116068 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:17.116342 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:17.116345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:17.116347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:17.116716 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:17.116956 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:17.117184 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 20:47:17.117417 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 20:47:17.123458 6484 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:17.121705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52864 (local address 127.0.97.1:36553)
0929 20:47:17.121887 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:17.121893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:17.122181 (+ 288us) server_negotiation.cc:407] Connection header received
0929 20:47:17.122325 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:17.122328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:17.122412 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:17.122517 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:17.123230 (+ 713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52864: BlockingRecv error: recv got EOF from 127.0.0.1:52864 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 20:47:17.123621 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/krb5cc' of type FILE
Sep 29 20:47:17 dist-test-slave-blv7 krb5kdc[5976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:17.125196 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 20:47:17 dist-test-slave-blv7 krb5kdc[5976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:17.126614 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 20:47:17 dist-test-slave-blv7 krb5kdc[5976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:47:17.140226 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:17.127948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52876 (local address 127.0.97.1:36553)
0929 20:47:17.128099 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:17.128105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:17.128473 (+ 368us) server_negotiation.cc:407] Connection header received
0929 20:47:17.128614 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:17.128618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:17.128679 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:17.128774 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:17.129627 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:17.130572 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:17.131858 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:17.132253 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:17.132340 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:17.135623 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:17.135628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:17.135631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:17.135663 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:17.137927 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:17.138496 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:17.138501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:17.138504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:17.138607 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:17.138917 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:17.138921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:17.138923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:17.139302 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:17.139513 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:17.139708 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 20:47:17.139884 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 20:47:17.144328 6484 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:17.142561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52882 (local address 127.0.97.1:36553)
0929 20:47:17.142727 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:17.142733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:17.142978 (+ 245us) server_negotiation.cc:407] Connection header received
0929 20:47:17.143159 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:17.143164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:17.143234 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:17.143317 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:17.144117 (+ 800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52882: BlockingRecv error: recv got EOF from 127.0.0.1:52882 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 20:47:17.144452 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790714786636870-388-0/krb5kdc/krb5cc' of type FILE
Sep 29 20:47:17 dist-test-slave-blv7 krb5kdc[5976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:17.145819 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 20:47:17 dist-test-slave-blv7 krb5kdc[5976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:17.147002 388 init.cc:295] Successfully reacquired a new kerberos TGT
I20260929 20:47:17.151875 6210 heartbeater.cc:503] Master 127.0.97.62:35225 was elected leader, sending a full tablet report...
Sep 29 20:47:17 dist-test-slave-blv7 krb5kdc[5976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:47:17.160070 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:17.148309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.0.97.1:36553)
0929 20:47:17.148461 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:17.148464 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:17.148619 (+ 155us) server_negotiation.cc:407] Connection header received
0929 20:47:17.148736 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:17.148739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:17.148805 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:17.148868 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:17.149764 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:17.150427 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:17.151616 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:17.152213 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:17.152303 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:17.155739 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:17.155744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:17.155747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:17.155775 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:17.158076 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:17.158624 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:17.158627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:17.158629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:17.158738 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:17.159044 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:17.159048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:17.159051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:17.159299 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:17.159510 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:17.159675 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 20:47:17.159829 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:47:17.161427 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 6082
I20260929 20:47:17.172616 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:17.162198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52900 (local address 127.0.97.1:36553)
0929 20:47:17.162377 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:17.162382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:17.162561 (+ 179us) server_negotiation.cc:407] Connection header received
0929 20:47:17.162676 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:17.162680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:17.162734 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:17.162816 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:17.163688 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:17.164366 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:17.165525 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:17.165963 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:17.166077 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:17.167904 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:17.167909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:17.167912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:17.167942 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:17.170412 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:17.170977 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:17.171041 (+ 64us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:17.171046 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:17.171192 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:17.171457 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:17.171460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:17.171468 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:17.171712 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:17.171909 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:17.172138 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 20:47:17.172305 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:47:17.173174 6205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:17.252620 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 6082
I20260929 20:47:17.265159 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 6216
Sep 29 20:47:17 dist-test-slave-blv7 krb5kdc[5976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:47:17.278960 6497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:17.265935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55242 (local address 127.0.97.2:33389)
0929 20:47:17.266448 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:17.266455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:17.266479 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:47:17.266557 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:17.266564 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:17.266770 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:17.266901 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:17.267768 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:17.268797 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:17.269904 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:17.270434 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:17.270533 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:17.273835 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:17.273846 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:17.273860 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:17.273891 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:17.276622 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:17.277179 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:17.277187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:17.277189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:17.277325 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:17.277611 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:17.277614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:17.277616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:17.277974 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:17.278219 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:17.278414 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 20:47:17.278635 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":258,"threads_started":1}
I20260929 20:47:17.280237 6339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:17.359596 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 6216
I20260929 20:47:17.371615 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 6349
Sep 29 20:47:17 dist-test-slave-blv7 krb5kdc[5976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:47:17.386442 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:17.372695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42056 (local address 127.0.97.3:34687)
0929 20:47:17.373391 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:17.373398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:17.373429 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:47:17.373498 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:17.373505 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:17.373722 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:17.373857 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:17.374758 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:17.375931 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:17.377106 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:17.377696 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:17.377814 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:17.381464 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:17.381474 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:17.381489 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:17.381528 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:17.384150 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:17.384686 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:17.384693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:17.384695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:17.384833 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:17.385102 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:17.385105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:17.385106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:17.385414 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:17.385639 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:17.385861 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 20:47:17.386153 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":367,"threads_started":1}
I20260929 20:47:17.387727 6472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:17.468820 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 6349
I20260929 20:47:17.480849 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 5989
I20260929 20:47:17.481946 6051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:17.578543 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 5989
2026-09-29T20:47:17Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3549 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:47:17 dist-test-slave-blv7 krb5kdc[6504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:47:17 dist-test-slave-blv7 krb5kdc[6504](info): set up 2 sockets
Sep 29 20:47:17 dist-test-slave-blv7 krb5kdc[6504](info): commencing operation
krb5kdc: starting...
W20260929 20:47:18.567420 388 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.008s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:47:18 dist-test-slave-blv7 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:47:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:47:18Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260929 20:47:18.748023 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45167
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:19.045116 6517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:19.045380 6517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:19.045503 6517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:19.052493 6517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:47:19.052589 6517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:19.052640 6517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:19.052714 6517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:47:19.052758 6517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:47:19.062350 6517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:19.063632 6517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:19.064874 6517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:19.075521 6523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:19.075555 6525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:19.075579 6522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:19.076195 6517 server_base.cc:1034] running on GCE node
I20260929 20:47:19.076856 6517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:19.078087 6517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:19.079283 6517 hybrid_clock.cc:648] HybridClock initialized: now 1790714839079236 us; error 36 us; skew 500 ppm
I20260929 20:47:19.079572 6517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:19 dist-test-slave-blv7 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:19.082418 6517 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260929 20:47:19.083783 6517 webserver.cc:460] Webserver started at http://127.0.97.62:38009/ using document root <none> and password file <none>
I20260929 20:47:19.084405 6517 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:19.084513 6517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:19.084776 6517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:19.087544 6517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "1460986649014da8aff571f52348fcad"
format_stamp: "Formatted at 2026-09-29 20:47:19 on dist-test-slave-blv7"
server_key: "f38e2161edd271a9eda8841407db8cbc"
server_key_iv: "08d5f1eb1404daa2265dbde670dc551e"
server_key_version: "encryptionkey@0"
I20260929 20:47:19.088238 6517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "1460986649014da8aff571f52348fcad"
format_stamp: "Formatted at 2026-09-29 20:47:19 on dist-test-slave-blv7"
server_key: "f38e2161edd271a9eda8841407db8cbc"
server_key_iv: "08d5f1eb1404daa2265dbde670dc551e"
server_key_version: "encryptionkey@0"
I20260929 20:47:19.092873 6517 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:47:19.096009 6532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:19.097246 6517 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 20:47:19.097386 6517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "1460986649014da8aff571f52348fcad"
format_stamp: "Formatted at 2026-09-29 20:47:19 on dist-test-slave-blv7"
server_key: "f38e2161edd271a9eda8841407db8cbc"
server_key_iv: "08d5f1eb1404daa2265dbde670dc551e"
server_key_version: "encryptionkey@0"
I20260929 20:47:19.097544 6517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:19.117056 6517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:19.121017 6517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:19.121263 6517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:19.137207 6517 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45167
I20260929 20:47:19.137226 6583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45167 every 8 connection(s)
I20260929 20:47:19.138463 6517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:47:19.141904 6584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:19.143736 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 6517
I20260929 20:47:19.143930 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:47:19.144692 388 external_mini_cluster.cc:1428] Setting key d9a40b4bc7f85b83c782ae3e2df1a696
I20260929 20:47:19.153674 6584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad: Bootstrap starting.
I20260929 20:47:19.157213 6584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad: Neither blocks nor log segments found. Creating new log.
Sep 29 20:47:19 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:47:19.158315 6584 log.cc:824] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:19.161127 6584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad: No bootstrap required, opened a new log
I20260929 20:47:19.165977 6587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:19.146645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46186 (local address 127.0.97.62:45167)
0929 20:47:19.147417 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:19.147428 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:19.147466 (+ 38us) server_negotiation.cc:407] Connection header received
0929 20:47:19.149013 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:19.149033 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:19.149397 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:19.149695 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:19.151385 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:19.152660 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:19.154305 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:19.154932 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:19.155100 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:19.159421 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:19.159432 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:19.159449 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:19.159485 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:19.162340 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:19.163091 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:19.163097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:19.163102 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:19.163250 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:19.163598 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:19.163601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:19.163603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:19.164071 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:19.164282 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:19.164689 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 20:47:19.164995 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":348,"threads_started":1}
I20260929 20:47:19.169049 6584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } }
I20260929 20:47:19.169382 6584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:19.169464 6584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1460986649014da8aff571f52348fcad, State: Initialized, Role: FOLLOWER
I20260929 20:47:19.169971 6584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } }
I20260929 20:47:19.170169 6584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:47:19.170285 6584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:47:19.170416 6584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:19.172647 6584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } }
I20260929 20:47:19.173075 6584 leader_election.cc:302] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1460986649014da8aff571f52348fcad; no voters:
I20260929 20:47:19.173591 6584 leader_election.cc:288] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:47:19.173712 6589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:47:19.174366 6589 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [term 1 LEADER]: Becoming Leader. State: Replica: 1460986649014da8aff571f52348fcad, State: Running, Role: LEADER
I20260929 20:47:19.174870 6589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } }
I20260929 20:47:19.175506 6584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:47:19.177134 6590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1460986649014da8aff571f52348fcad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } }
I20260929 20:47:19.177448 6590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:19.178870 6594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:47:19.178727 6591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1460986649014da8aff571f52348fcad. Latest consensus state: current_term: 1 leader_uuid: "1460986649014da8aff571f52348fcad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } }
I20260929 20:47:19.179275 6591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:19.184742 6594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:47:19.192708 6594 catalog_manager.cc:1347] Generated new cluster ID: 48312dabe6844bb0a0147505a1576ca1
I20260929 20:47:19.192811 6594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:47:19.209153 6594 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:47:19.210891 6594 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:47:19.224071 6594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad: Generated new TSK 0
I20260929 20:47:19.225055 6594 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260929 20:47:19.268677 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45167
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:19.572911 6610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:19.573165 6610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:19.573279 6610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:19.580139 6610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:19.580228 6610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:19.580366 6610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:47:19.589506 6610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:19.590586 6610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:19.591768 6610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:19.603731 6615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:19.603796 6616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:19.603766 6618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:19.604413 6610 server_base.cc:1034] running on GCE node
I20260929 20:47:19.604893 6610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:19.605702 6610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:19.606884 6610 hybrid_clock.cc:648] HybridClock initialized: now 1790714839606828 us; error 48 us; skew 500 ppm
I20260929 20:47:19.607190 6610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:19 dist-test-slave-blv7 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:19.610024 6610 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 20:47:19.611397 6610 webserver.cc:460] Webserver started at http://127.0.97.1:46629/ using document root <none> and password file <none>
I20260929 20:47:19.612011 6610 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:19.612126 6610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:19.612383 6610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:19.615166 6610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "d343b75e87f142f29eb74970160c325e"
format_stamp: "Formatted at 2026-09-29 20:47:19 on dist-test-slave-blv7"
server_key: "b9f713e1aa084cf26417e50b85c34946"
server_key_iv: "23a7426c929c8c94891165d7fe00ed4f"
server_key_version: "encryptionkey@0"
I20260929 20:47:19.615863 6610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d343b75e87f142f29eb74970160c325e"
format_stamp: "Formatted at 2026-09-29 20:47:19 on dist-test-slave-blv7"
server_key: "b9f713e1aa084cf26417e50b85c34946"
server_key_iv: "23a7426c929c8c94891165d7fe00ed4f"
server_key_version: "encryptionkey@0"
I20260929 20:47:19.620452 6610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:47:19.623584 6625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:19.624812 6610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:47:19.624945 6610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "d343b75e87f142f29eb74970160c325e"
format_stamp: "Formatted at 2026-09-29 20:47:19 on dist-test-slave-blv7"
server_key: "b9f713e1aa084cf26417e50b85c34946"
server_key_iv: "23a7426c929c8c94891165d7fe00ed4f"
server_key_version: "encryptionkey@0"
I20260929 20:47:19.625115 6610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:19.637879 6610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:19.641513 6610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:19.641757 6610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:19.642586 6610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:19.643761 6610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:19.643833 6610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:19.643946 6610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:19.644009 6610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:19.672842 6610 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41329
I20260929 20:47:19.672884 6737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41329 every 8 connection(s)
I20260929 20:47:19.674037 6610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:47:19.677776 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 6610
I20260929 20:47:19.677968 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:47:19.678736 388 external_mini_cluster.cc:1428] Setting key 93dd39cb802266d84e3dcf21afe9636c
Sep 29 20:47:19 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:47:19.694022 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:19.676410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42003 (local address 127.0.97.62:45167)
0929 20:47:19.676764 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:19.676768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:19.679079 (+ 2311us) server_negotiation.cc:407] Connection header received
0929 20:47:19.680072 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:19.680076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:19.680140 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:19.680245 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:19.682177 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:19.682869 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:19.684826 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:19.685240 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:19.685356 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:19.689010 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:19.689015 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:19.689018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:19.689048 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:19.691263 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:19.692009 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:19.692012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:19.692014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:19.692130 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:19.692538 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:19.692541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:19.692543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:19.692775 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:19.692870 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:19.693627 (+ 757us) server_negotiation.cc:299] Negotiation successful
0929 20:47:19.693783 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":178,"threads_started":1}
I20260929 20:47:19.695426 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:19.676936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.97.1:42003)
0929 20:47:19.677725 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:19.677775 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:19.679794 (+ 2019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:19.680531 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:19.680539 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:19.681242 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:19.682018 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:19.682042 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:19.683009 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:19.683035 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:19.684707 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:19.684719 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:19.685453 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:19.685459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:19.685716 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:19.686365 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:19.686391 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:19.688809 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:19.691390 (+ 2581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:19.691399 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:19.691415 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:19.691913 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:19.692229 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:19.692233 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:19.692236 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:19.692450 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:19.692880 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:19.692887 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:19.693288 (+ 401us) client_negotiation.cc:769] Sending connection context
0929 20:47:19.693568 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 20:47:19.694052 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":278,"threads_started":1}
I20260929 20:47:19.696846 6738 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45167
I20260929 20:47:19.697188 6738 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:19.698027 6738 heartbeater.cc:511] Master 127.0.97.62:45167 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:47:19.700172 6549 ts_manager.cc:194] Registered new tserver with Master: d343b75e87f142f29eb74970160c325e (127.0.97.1:41329)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260929 20:47:19.701870 6549 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260929 20:47:19.722285 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45167
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:20.021488 6744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:20.021755 6744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:20.021876 6744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:20.028898 6744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:20.029007 6744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:20.029168 6744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:47:20.038335 6744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:20.039551 6744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:20.040800 6744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:20.052510 6750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:20.052538 6749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:20.053009 6752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:20.053238 6744 server_base.cc:1034] running on GCE node
I20260929 20:47:20.053763 6744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:20.054602 6744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:20.055776 6744 hybrid_clock.cc:648] HybridClock initialized: now 1790714840055725 us; error 48 us; skew 500 ppm
I20260929 20:47:20.056056 6744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:20 dist-test-slave-blv7 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714840, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:20.058948 6744 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 20:47:20.060330 6744 webserver.cc:460] Webserver started at http://127.0.97.2:32963/ using document root <none> and password file <none>
I20260929 20:47:20.060987 6744 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:20.061097 6744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:20.061372 6744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:20.064155 6744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "16a15205be8e418ba7b0856023a000ee"
format_stamp: "Formatted at 2026-09-29 20:47:20 on dist-test-slave-blv7"
server_key: "4c78c13440cbc1cc5d34f09eca573900"
server_key_iv: "994a7d49a06ed5ddf2d8a7106a2daf30"
server_key_version: "encryptionkey@0"
I20260929 20:47:20.064884 6744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "16a15205be8e418ba7b0856023a000ee"
format_stamp: "Formatted at 2026-09-29 20:47:20 on dist-test-slave-blv7"
server_key: "4c78c13440cbc1cc5d34f09eca573900"
server_key_iv: "994a7d49a06ed5ddf2d8a7106a2daf30"
server_key_version: "encryptionkey@0"
I20260929 20:47:20.069631 6744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:47:20.072865 6759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:20.074115 6744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:47:20.074262 6744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "16a15205be8e418ba7b0856023a000ee"
format_stamp: "Formatted at 2026-09-29 20:47:20 on dist-test-slave-blv7"
server_key: "4c78c13440cbc1cc5d34f09eca573900"
server_key_iv: "994a7d49a06ed5ddf2d8a7106a2daf30"
server_key_version: "encryptionkey@0"
I20260929 20:47:20.074424 6744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:20.089154 6744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:20.093027 6744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:20.093291 6744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:20.094156 6744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:20.095415 6744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:20.095489 6744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:20.095597 6744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:20.095651 6744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:20.124272 6744 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33431
I20260929 20:47:20.124313 6871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33431 every 8 connection(s)
I20260929 20:47:20.125556 6744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:47:20.127454 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 6744
I20260929 20:47:20.127583 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:47:20.128111 388 external_mini_cluster.cc:1428] Setting key 6652eb1e6ae1ebe6771edab4e07d132a
Sep 29 20:47:20 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714840, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:47:20.146649 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:20.127979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45649 (local address 127.0.97.62:45167)
0929 20:47:20.128164 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:20.128171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:20.129407 (+ 1236us) server_negotiation.cc:407] Connection header received
0929 20:47:20.131312 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:20.131318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:20.131393 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:20.131494 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:20.132845 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:20.133783 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:20.135456 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:20.135879 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:20.135979 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:20.140042 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:20.140048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:20.140053 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:20.140097 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:20.143347 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:20.144210 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:20.144216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:20.144220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:20.144383 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:20.144825 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:20.144830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:20.144833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:20.145179 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:20.145317 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:20.146106 (+ 789us) server_negotiation.cc:299] Negotiation successful
0929 20:47:20.146319 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:47:20.147794 6874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:20.128478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.97.2:45649)
0929 20:47:20.129243 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:20.129292 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:20.131063 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:20.131665 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:20.131672 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:20.132169 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:20.132699 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:20.132717 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:20.133937 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:20.133940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:20.135241 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:20.135255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:20.136100 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:20.136107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:20.136389 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:20.137104 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:20.137130 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:20.139820 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:20.143477 (+ 3657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:20.143486 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:20.143502 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:20.144067 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:20.144484 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:20.144489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:20.144493 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:20.144708 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:20.145291 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:20.145299 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:20.145682 (+ 383us) client_negotiation.cc:769] Sending connection context
0929 20:47:20.145964 (+ 282us) client_negotiation.cc:240] Negotiation successful
0929 20:47:20.146457 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":299,"threads_started":1}
I20260929 20:47:20.149174 6872 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45167
I20260929 20:47:20.149439 6872 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
I20260929 20:47:20.150179 6872 heartbeater.cc:511] Master 127.0.97.62:45167 requested a full tablet report, sending...
I20260929 20:47:20.151469 6549 ts_manager.cc:194] Registered new tserver with Master: 16a15205be8e418ba7b0856023a000ee (127.0.97.2:33431)
I20260929 20:47:20.152249 6549 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 20:47:20.174000 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45167
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:20.472477 6877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:20.472750 6877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:20.472873 6877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:20.480120 6877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:20.480238 6877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:20.480388 6877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:47:20.489743 6877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:20.490962 6877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:20.492233 6877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:20.504195 6882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:20.504274 6885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:20.504339 6883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:20.505069 6877 server_base.cc:1034] running on GCE node
I20260929 20:47:20.505577 6877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:20.506466 6877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:20.507691 6877 hybrid_clock.cc:648] HybridClock initialized: now 1790714840507650 us; error 43 us; skew 500 ppm
I20260929 20:47:20.507961 6877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:20 dist-test-slave-blv7 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714840, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:20.510931 6877 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 20:47:20.512441 6877 webserver.cc:460] Webserver started at http://127.0.97.3:45121/ using document root <none> and password file <none>
I20260929 20:47:20.513049 6877 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:20.513154 6877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:20.513423 6877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:20.516316 6877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "dd36b520f02e4c7fa3082e278f14b414"
format_stamp: "Formatted at 2026-09-29 20:47:20 on dist-test-slave-blv7"
server_key: "f5c371d0477c4cb20806a21e32e50692"
server_key_iv: "c6b125ad31a9740668f7721744108457"
server_key_version: "encryptionkey@0"
I20260929 20:47:20.517127 6877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "dd36b520f02e4c7fa3082e278f14b414"
format_stamp: "Formatted at 2026-09-29 20:47:20 on dist-test-slave-blv7"
server_key: "f5c371d0477c4cb20806a21e32e50692"
server_key_iv: "c6b125ad31a9740668f7721744108457"
server_key_version: "encryptionkey@0"
I20260929 20:47:20.521961 6877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:47:20.525245 6892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:20.526547 6877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:47:20.526710 6877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "dd36b520f02e4c7fa3082e278f14b414"
format_stamp: "Formatted at 2026-09-29 20:47:20 on dist-test-slave-blv7"
server_key: "f5c371d0477c4cb20806a21e32e50692"
server_key_iv: "c6b125ad31a9740668f7721744108457"
server_key_version: "encryptionkey@0"
I20260929 20:47:20.526873 6877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:20.555999 6877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:20.559901 6877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:20.560153 6877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:20.561089 6877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:20.562345 6877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:20.562415 6877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:20.562527 6877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:20.562583 6877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:20.590953 6877 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39649
I20260929 20:47:20.591001 7004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39649 every 8 connection(s)
I20260929 20:47:20.592171 6877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:47:20.599934 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 6877
I20260929 20:47:20.600099 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:47:20.600616 388 external_mini_cluster.cc:1428] Setting key dfe95bfa6d566698222c883418cf2cb8
Sep 29 20:47:20 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714840, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:47:20.610289 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:20.594477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41911 (local address 127.0.97.62:45167)
0929 20:47:20.594637 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:20.594641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:20.595783 (+ 1142us) server_negotiation.cc:407] Connection header received
0929 20:47:20.597474 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:20.597480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:20.597555 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:20.597639 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:20.599061 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:20.599858 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:20.601832 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:20.602222 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:20.602330 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:20.605449 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:20.605454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:20.605457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:20.605489 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:20.607821 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:20.608406 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:20.608409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:20.608411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:20.608518 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:20.608865 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:20.608868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:20.608870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:20.609097 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:20.609203 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:20.609910 (+ 707us) server_negotiation.cc:299] Negotiation successful
0929 20:47:20.610053 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:47:20.611299 7007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:20.594952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.97.3:41911)
0929 20:47:20.595629 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:20.595670 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:20.597222 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:20.597871 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:20.597879 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:20.598351 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:20.598884 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:20.598900 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:20.600032 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:20.600038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:20.601697 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:20.601711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:20.602429 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:20.602436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:20.602660 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:20.603316 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:20.603336 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:20.605268 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:20.607943 (+ 2675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:20.607949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:20.607961 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:20.608319 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:20.608604 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:20.608607 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:20.608609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:20.608787 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:20.609286 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:20.609292 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:20.609603 (+ 311us) client_negotiation.cc:769] Sending connection context
0929 20:47:20.609840 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 20:47:20.610223 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":255,"threads_started":1}
I20260929 20:47:20.612588 7005 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45167
I20260929 20:47:20.612830 7005 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:20.613453 7005 heartbeater.cc:511] Master 127.0.97.62:45167 requested a full tablet report, sending...
I20260929 20:47:20.614867 6549 ts_manager.cc:194] Registered new tserver with Master: dd36b520f02e4c7fa3082e278f14b414 (127.0.97.3:39649)
I20260929 20:47:20.615536 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:47:20.615962 6549 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41911
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260929 20:47:20.658984 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42803
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45167,127.0.97.61:42803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:47:20.704784 6738 heartbeater.cc:503] Master 127.0.97.62:45167 was elected leader, sending a full tablet report...
W20260929 20:47:20.960822 7010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:20.961089 7010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:20.961206 7010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:20.968070 7010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:47:20.968220 7010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:20.968303 7010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:20.968376 7010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:47:20.968433 7010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:47:20.977850 7010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45167,127.0.97.61:42803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
W20260929 20:47:20.978166 7010 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:45167,127.0.97.61:42803'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 20:47:20.979200 7010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:20.980406 7010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:20.990990 7015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:20.991168 7016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:20.991291 7018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:20.991783 7010 server_base.cc:1034] running on GCE node
I20260929 20:47:20.992467 7010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:20.993707 7010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:20.994921 7010 hybrid_clock.cc:648] HybridClock initialized: now 1790714840994854 us; error 45 us; skew 500 ppm
I20260929 20:47:20.995245 7010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:20 dist-test-slave-blv7 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714840, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:20.998178 7010 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260929 20:47:20.999509 7010 webserver.cc:460] Webserver started at http://127.0.97.61:33767/ using document root <none> and password file <none>
I20260929 20:47:21.000185 7010 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:21.000276 7010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:21.000564 7010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:21.003428 7010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/data/instance:
uuid: "1097f58cae6a4662999174443b82b300"
format_stamp: "Formatted at 2026-09-29 20:47:20 on dist-test-slave-blv7"
server_key: "ba7cdd51006afd363d34b32a225e1b84"
server_key_iv: "a50d14c17d71e9aba58ea300385f976d"
server_key_version: "encryptionkey@0"
I20260929 20:47:21.004127 7010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/wal/instance:
uuid: "1097f58cae6a4662999174443b82b300"
format_stamp: "Formatted at 2026-09-29 20:47:20 on dist-test-slave-blv7"
server_key: "ba7cdd51006afd363d34b32a225e1b84"
server_key_iv: "a50d14c17d71e9aba58ea300385f976d"
server_key_version: "encryptionkey@0"
I20260929 20:47:21.008823 7010 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 20:47:21.011899 7025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:21.013175 7010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:47:21.013315 7010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/wal
uuid: "1097f58cae6a4662999174443b82b300"
format_stamp: "Formatted at 2026-09-29 20:47:20 on dist-test-slave-blv7"
server_key: "ba7cdd51006afd363d34b32a225e1b84"
server_key_iv: "a50d14c17d71e9aba58ea300385f976d"
server_key_version: "encryptionkey@0"
I20260929 20:47:21.013486 7010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:21.029521 7010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:21.033288 7010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:21.033505 7010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:21.049345 7010 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:42803
I20260929 20:47:21.049317 7076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:42803 every 8 connection(s)
I20260929 20:47:21.050534 7010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/data/info.pb
I20260929 20:47:21.053982 7077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:21.055567 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 7010
I20260929 20:47:21.063074 7077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:47:21 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 29 20:47:21 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714840, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:47:21.077719 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.057617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40522 (local address 127.0.97.61:42803)
0929 20:47:21.058484 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:21.058499 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:21.058863 (+ 364us) server_negotiation.cc:407] Connection header received
0929 20:47:21.060472 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:21.060494 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:21.060903 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:21.061201 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:21.062853 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.064198 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.065662 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.066615 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.066766 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.070634 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:21.070644 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:21.070654 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:21.070686 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:21.073517 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.074219 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.074227 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.074237 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.074414 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.074796 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.074800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.074803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.075490 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:21.075746 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:21.076110 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 20:47:21.076504 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":368,"threads_started":1}
I20260929 20:47:21.079487 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.065588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46188 (local address 127.0.97.62:45167)
0929 20:47:21.065791 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:21.065797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:21.066333 (+ 536us) server_negotiation.cc:407] Connection header received
0929 20:47:21.066488 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:21.066493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:21.066551 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:21.066630 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:21.067798 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.068460 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.070007 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.070405 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.070506 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.073792 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:21.073796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:21.073800 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:21.073830 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:21.076873 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.077482 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.077487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.077490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.077647 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.078062 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.078067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.078071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.078355 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:21.078478 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:21.078984 (+ 506us) server_negotiation.cc:299] Negotiation successful
0929 20:47:21.079200 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:47:21.079551 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.065529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.0.1:46188)
0929 20:47:21.066204 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:21.066236 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:21.066381 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:21.066717 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:21.066725 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:21.067398 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:21.067660 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.067668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.068589 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.068592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.069882 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.069893 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.070632 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.070640 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.070779 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.071391 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:21.071416 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:21.073609 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:21.076986 (+ 3377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.076994 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.076999 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.077373 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.077765 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.077769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.077798 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.077947 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.078480 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:21.078487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:21.078636 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 20:47:21.078848 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 20:47:21.079149 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":364,"threads_started":1}
I20260929 20:47:21.081067 7077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:47:21 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714840, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 20:47:21.091406 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.081623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42803 (local address 127.0.0.1:40526)
0929 20:47:21.081803 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:21.081824 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:21.081910 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:21.082153 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:21.082156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:21.082445 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:21.082566 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.082573 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.083473 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.083476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.084296 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.084305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.084908 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.084911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.085032 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.085464 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:21.085479 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:21.087299 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:21.089623 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.089626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.089629 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.089933 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.090193 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.090196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.090198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.090306 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.090721 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:21.090724 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:21.090825 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 20:47:21.090932 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 20:47:21.091163 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 20:47:21.091544 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.081832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40526 (local address 127.0.97.61:42803)
0929 20:47:21.081949 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:21.081953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:21.081965 (+ 12us) server_negotiation.cc:407] Connection header received
0929 20:47:21.081997 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:21.082000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:21.082046 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:21.082122 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:21.082668 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.083374 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.084416 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.084799 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.084887 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.087458 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:21.087462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:21.087466 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:21.087491 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:21.089503 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.090018 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.090021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.090023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.090116 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.090385 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.090388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.090390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.090635 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:21.090731 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:21.091004 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 20:47:21.091276 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":44}
I20260929 20:47:21.095758 7077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300: Bootstrap starting.
I20260929 20:47:21.098547 7077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:21.099514 7077 log.cc:824] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:21.101948 7077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300: No bootstrap required, opened a new log
I20260929 20:47:21.108457 7077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } }
I20260929 20:47:21.108749 7077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:21.108831 7077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1097f58cae6a4662999174443b82b300, State: Initialized, Role: FOLLOWER
I20260929 20:47:21.109403 7077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } }
I20260929 20:47:21.110133 7083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } } }
I20260929 20:47:21.110391 7083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:47:21.111068 7077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 20:47:21.118657 7094 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:47:21.118760 7094 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:47:21.121285 7010 master_runner.cc:428] Detected that this master 1097f58cae6a4662999174443b82b300 is joining an existing cluster
I20260929 20:47:21.121371 7010 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:42803
I20260929 20:47:21.139899 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.124881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46198 (local address 127.0.97.62:45167)
0929 20:47:21.125044 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:21.125050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:21.125683 (+ 633us) server_negotiation.cc:407] Connection header received
0929 20:47:21.125909 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:21.125914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:21.126001 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:21.126067 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:21.127433 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.128511 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.130376 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.130961 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.131061 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.133402 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:21.133409 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:21.133414 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:21.133451 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:21.136658 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.137583 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.137589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.137592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.137760 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.138209 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.138214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.138218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.138564 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:21.138697 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:21.139320 (+ 623us) server_negotiation.cc:299] Negotiation successful
0929 20:47:21.139540 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:47:21.141175 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.124950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.0.1:46198)
0929 20:47:21.125581 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:21.125600 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:21.125778 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:21.126258 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:21.126262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:21.126873 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:21.127265 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.127277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.128670 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.128675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.130205 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.130219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.131330 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.131336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.131510 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.132160 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:21.132182 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:21.133213 (+ 1031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:21.136799 (+ 3586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.136804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.136808 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.137457 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.137905 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.137910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.137913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.138068 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.138798 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:21.138803 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:21.138990 (+ 187us) client_negotiation.cc:769] Sending connection context
0929 20:47:21.140390 (+ 1400us) client_negotiation.cc:240] Negotiation successful
0929 20:47:21.140698 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":341,"threads_started":1}
I20260929 20:47:21.154549 6872 heartbeater.cc:503] Master 127.0.97.62:45167 was elected leader, sending a full tablet report...
I20260929 20:47:21.157361 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.125802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.0.97.61:42803)
0929 20:47:21.125968 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:21.125973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:21.130561 (+ 4588us) server_negotiation.cc:407] Connection header received
0929 20:47:21.130739 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:21.130744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:21.130800 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:21.130859 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:21.132203 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.133270 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.135568 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.136291 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.136384 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.144373 (+ 7989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:21.144379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:21.144384 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:21.144415 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:21.147660 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.154846 (+ 7186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.154852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.154855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.154989 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.155637 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.155642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.155645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.155994 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:21.156129 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:21.156824 (+ 695us) server_negotiation.cc:299] Negotiation successful
0929 20:47:21.157034 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 20:47:21.158576 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.126337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42803 (local address 127.0.0.1:40534)
0929 20:47:21.130491 (+ 4154us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:21.130507 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:21.130639 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:21.131099 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:21.131103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:21.131761 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:21.132056 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.132069 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.133445 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.133450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.134942 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.134955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.136666 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.136672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.136826 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.143194 (+ 6368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:21.143219 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:21.144207 (+ 988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:21.147794 (+ 3587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.147799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.147803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.154304 (+ 6501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.155147 (+ 843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.155151 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.155155 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.155298 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.156201 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:21.156205 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:21.156329 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 20:47:21.157898 (+ 1569us) client_negotiation.cc:240] Negotiation successful
0929 20:47:21.158167 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4040,"thread_start_us":3863,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 29 20:47:21 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 20:47:21.177484 7103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.147809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42803 (local address 127.0.0.1:40544)
0929 20:47:21.163375 (+ 15566us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:21.163414 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:21.163603 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:21.164029 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:21.164037 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:21.164891 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:21.165157 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.165169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.166352 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.166357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.167696 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.167708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.168498 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.168505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.168706 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.169357 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:21.169379 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:21.171387 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:21.174574 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.174586 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.174593 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.175141 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.175557 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.175562 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.175566 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.175740 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.176446 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:21.176454 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:21.176635 (+ 181us) client_negotiation.cc:769] Sending connection context
0929 20:47:21.176807 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 20:47:21.177094 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15381,"thread_start_us":15258,"threads_started":1}
I20260929 20:47:21.177588 7080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.148147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40544 (local address 127.0.97.61:42803)
0929 20:47:21.157858 (+ 9711us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:21.157867 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:21.163508 (+ 5641us) server_negotiation.cc:407] Connection header received
0929 20:47:21.163797 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:21.163801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:21.163878 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:21.163981 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:21.165293 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.166174 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.167825 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.168280 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.168477 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.171556 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:21.171561 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:21.171564 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:21.171591 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:21.174451 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.175290 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.175296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.175300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.175449 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.175869 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.175874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.175878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.176291 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:21.176430 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:21.176989 (+ 559us) server_negotiation.cc:299] Negotiation successful
0929 20:47:21.177219 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9620,"thread_start_us":14741,"threads_started":1}
I20260929 20:47:21.179663 6549 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:42803: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1097f58cae6a4662999174443b82b300"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 42803
}
attrs {
promote: true
}
}
dest_uuid: "1460986649014da8aff571f52348fcad"
cas_config_opid_index: -1
I20260929 20:47:21.180775 6549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: true } }
I20260929 20:47:21.192263 7052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:21.191668 7107 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1097f58cae6a4662999174443b82b300 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260929 20:47:21.194905 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46467
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45167,127.0.97.61:42803,127.0.97.60:46467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:47:21.195910 7107 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:42803
I20260929 20:47:21.195899 7052 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 1 FOLLOWER]: Refusing update from remote peer 1460986649014da8aff571f52348fcad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 20:47:21.196563 7010 master.cc:537] Master@127.0.97.61:42803 shutting down...
I20260929 20:47:21.202116 7109 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:47:21.202510 7108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1460986649014da8aff571f52348fcad" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: true } } }
I20260929 20:47:21.202847 7108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [sys.catalog]: This master's current role is: LEADER
W20260929 20:47:21.205094 6535 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 20:47:21.207726 6535 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad -> Peer 1097f58cae6a4662999174443b82b300 (127.0.97.61:42803): Couldn't send request to peer 1097f58cae6a4662999174443b82b300. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 20:47:21.209681 7010 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:47:21.210068 7010 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:21.210255 7010 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300: stopping tablet replica
I20260929 20:47:21.217001 7010 master.cc:552] Master@127.0.97.61:42803 shutdown complete.
I20260929 20:47:21.217088 7010 master_runner.cc:305] Clearing existing system tablet
I20260929 20:47:21.217533 7010 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:21.217657 7010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:21.220528 7010 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 20:47:21.222234 7114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:21.222652 7010 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:21.222900 7010 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 20:47:21.223043 7010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/wal
uuid: "1097f58cae6a4662999174443b82b300"
format_stamp: "Formatted at 2026-09-29 20:47:20 on dist-test-slave-blv7"
server_key: "ba7cdd51006afd363d34b32a225e1b84"
server_key_iv: "a50d14c17d71e9aba58ea300385f976d"
server_key_version: "encryptionkey@0"
I20260929 20:47:21.227906 7010 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:21.234786 7010 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 20:47:21.234987 7010 log.cc:1196] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 20:47:21.235486 7010 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300: Deleting consensus metadata
I20260929 20:47:21.235764 7010 master_runner.cc:315] Copying system tablet from 127.0.97.62:45167
I20260929 20:47:21.240399 7010 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:45167
I20260929 20:47:21.257329 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.241255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46212 (local address 127.0.97.62:45167)
0929 20:47:21.241426 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:21.241433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:21.243518 (+ 2085us) server_negotiation.cc:407] Connection header received
0929 20:47:21.243710 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:21.243715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:21.243805 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:21.243913 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:21.244945 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.246067 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.247888 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.248549 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.248682 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.251232 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:21.251240 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:21.251245 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:21.251284 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:21.254492 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.255308 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.255314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.255317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.255475 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.255873 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.255878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.255881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.256231 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:21.256378 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:21.256801 (+ 423us) server_negotiation.cc:299] Negotiation successful
0929 20:47:21.257015 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:47:21.257428 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.241207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.0.1:46212)
0929 20:47:21.243384 (+ 2177us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:21.243409 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:21.243569 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:21.243957 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:21.243962 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:21.244495 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:21.244787 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.244798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.246228 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.246233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.247724 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.247738 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.248713 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.248719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.248902 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.249977 (+ 1075us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:21.250004 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:21.250987 (+ 983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:21.254631 (+ 3644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.254636 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.254641 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.255177 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.255585 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.255590 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.255594 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.255756 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.256356 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:21.256361 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:21.256528 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 20:47:21.256680 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 20:47:21.256981 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2037,"thread_start_us":1946,"threads_started":1}
I20260929 20:47:21.258498 6569 tablet_copy_service.cc:140] P 1460986649014da8aff571f52348fcad: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1097f58cae6a4662999174443b82b300 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:46212)
I20260929 20:47:21.258775 6569 tablet_copy_service.cc:161] P 1460986649014da8aff571f52348fcad: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1097f58cae6a4662999174443b82b300 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:46212: session id = 1097f58cae6a4662999174443b82b300-00000000000000000000000000000000
I20260929 20:47:21.262956 6569 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 20:47:21.265198 7010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 20:47:21.271657 7010 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300: tablet copy: Starting download of 0 data blocks...
I20260929 20:47:21.271940 7010 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300: tablet copy: Starting download of 1 WAL segments...
I20260929 20:47:21.274529 7010 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 20:47:21.278750 6569 tablet_copy_service.cc:342] P 1460986649014da8aff571f52348fcad: Request end of tablet copy session 1097f58cae6a4662999174443b82b300-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:46212
I20260929 20:47:21.278949 6569 tablet_copy_service.cc:433] P 1460986649014da8aff571f52348fcad: ending tablet copy session 1097f58cae6a4662999174443b82b300-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1097f58cae6a4662999174443b82b300
I20260929 20:47:21.290572 7010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:21.293373 7121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:21.303524 7124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:21.306383 7122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:21.307142 7010 server_base.cc:1034] running on GCE node
I20260929 20:47:21.307463 7010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:21.308063 7010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:21.309365 7010 hybrid_clock.cc:648] HybridClock initialized: now 1790714841309307 us; error 38 us; skew 500 ppm
I20260929 20:47:21.309577 7010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:21 dist-test-slave-blv7 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714841, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:21.311722 7010 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260929 20:47:21.312978 7010 webserver.cc:460] Webserver started at http://127.0.97.61:34567/ using document root <none> and password file <none>
I20260929 20:47:21.313441 7010 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:21.313527 7010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:21.316478 7010 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260929 20:47:21.318269 7131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:21.318713 7010 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260929 20:47:21.318887 7010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/wal
uuid: "1097f58cae6a4662999174443b82b300"
format_stamp: "Formatted at 2026-09-29 20:47:20 on dist-test-slave-blv7"
server_key: "ba7cdd51006afd363d34b32a225e1b84"
server_key_iv: "a50d14c17d71e9aba58ea300385f976d"
server_key_version: "encryptionkey@0"
I20260929 20:47:21.319160 7010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:21.331984 7010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:21.335981 7010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:21.361516 7010 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:42803
I20260929 20:47:21.361562 7182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:42803 every 8 connection(s)
I20260929 20:47:21.362501 7010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-1/data/info.pb
I20260929 20:47:21.367657 7183 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:47:21.371690 7183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300: Bootstrap starting.
I20260929 20:47:21.387854 7183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:47:21.388320 7183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300: Bootstrap complete.
I20260929 20:47:21.389297 7183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: true } }
I20260929 20:47:21.389534 7183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1097f58cae6a4662999174443b82b300, State: Initialized, Role: LEARNER
I20260929 20:47:21.389799 7183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: true } }
I20260929 20:47:21.390507 7186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: true } } }
I20260929 20:47:21.390764 7183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:47:21.390874 7186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [sys.catalog]: This master's current role is: LEARNER
I20260929 20:47:21.397969 7197 catalog_manager.cc:1259] Loaded cluster ID: 48312dabe6844bb0a0147505a1576ca1
I20260929 20:47:21.398062 7197 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300: loading cluster ID for follower catalog manager: success
I20260929 20:47:21.401326 7197 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300: acquiring CA information for follower catalog manager: success
I20260929 20:47:21.403576 7197 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 20:47:21.534883 7110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:21.535188 7110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:21.535322 7110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:21.542219 7110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:47:21.542330 7110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:21.542403 7110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:21.542475 7110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:47:21.542532 7110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:47:21.552248 7110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45167,127.0.97.61:42803,127.0.97.60:46467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:21.553411 7110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:21.554612 7110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:21.565233 7202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:21.565274 7203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:21.565320 7205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:21.566382 7110 server_base.cc:1034] running on GCE node
I20260929 20:47:21.567165 7110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:21.568428 7110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:21.569625 7110 hybrid_clock.cc:648] HybridClock initialized: now 1790714841569574 us; error 40 us; skew 500 ppm
I20260929 20:47:21.569926 7110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:21 dist-test-slave-blv7 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714841, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:21.576879 7110 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260929 20:47:21.578455 7110 webserver.cc:460] Webserver started at http://127.0.97.60:46171/ using document root <none> and password file <none>
I20260929 20:47:21.579107 7110 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:21.579207 7110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:21.579439 7110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:21.582288 7110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/data/instance:
uuid: "ee15b91d5caa47b6b8021324f9fc53b1"
format_stamp: "Formatted at 2026-09-29 20:47:21 on dist-test-slave-blv7"
server_key: "95ca8ae0253b1d41cad10fdb14690203"
server_key_iv: "46f61bf642cef0e845f9dc7a28a73ccb"
server_key_version: "encryptionkey@0"
I20260929 20:47:21.582986 7110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/wal/instance:
uuid: "ee15b91d5caa47b6b8021324f9fc53b1"
format_stamp: "Formatted at 2026-09-29 20:47:21 on dist-test-slave-blv7"
server_key: "95ca8ae0253b1d41cad10fdb14690203"
server_key_iv: "46f61bf642cef0e845f9dc7a28a73ccb"
server_key_version: "encryptionkey@0"
I20260929 20:47:21.588003 7110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:47:21.591728 7212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:21.593027 7110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:47:21.593164 7110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/wal
uuid: "ee15b91d5caa47b6b8021324f9fc53b1"
format_stamp: "Formatted at 2026-09-29 20:47:21 on dist-test-slave-blv7"
server_key: "95ca8ae0253b1d41cad10fdb14690203"
server_key_iv: "46f61bf642cef0e845f9dc7a28a73ccb"
server_key_version: "encryptionkey@0"
I20260929 20:47:21.593348 7110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:21.606333 7110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:21.609997 7110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:21.610229 7110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:21.618578 7005 heartbeater.cc:503] Master 127.0.97.62:45167 was elected leader, sending a full tablet report...
I20260929 20:47:21.626777 7263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:46467 every 8 connection(s)
I20260929 20:47:21.626787 7110 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:46467
I20260929 20:47:21.628126 7110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/data/info.pb
I20260929 20:47:21.628522 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 7110
I20260929 20:47:21.632720 7264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:21.640790 7264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:47:21 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 20:47:21 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714841, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:47:21.651746 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.630778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38832 (local address 127.0.97.60:46467)
0929 20:47:21.631546 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:21.631560 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:21.631605 (+ 45us) server_negotiation.cc:407] Connection header received
0929 20:47:21.634163 (+ 2558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:21.634189 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:21.634645 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:21.635085 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:21.636328 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.638335 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.639580 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.640523 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.640694 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.644819 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:21.644827 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:21.644837 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:21.644879 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:21.647651 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.648409 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.648417 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.648425 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.648605 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.648979 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.648983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.648985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.649545 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:21.649761 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:21.650165 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 20:47:21.650532 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":331,"threads_started":1}
I20260929 20:47:21.657207 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.643226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46226 (local address 127.0.97.62:45167)
0929 20:47:21.643388 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:21.643395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:21.644353 (+ 958us) server_negotiation.cc:407] Connection header received
0929 20:47:21.644509 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:21.644514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:21.644580 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:21.644661 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:21.645745 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.646577 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.648167 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.648722 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.648812 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.652249 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:21.652255 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:21.652260 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:21.652299 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:21.654759 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.655424 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.655428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.655430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.655537 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.655866 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.655869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.655871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.656102 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:21.656274 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:21.656682 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 20:47:21.656889 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:47:21.657244 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.643112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.0.1:46226)
0929 20:47:21.644237 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:21.644269 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:21.644399 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:21.644726 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:21.644732 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:21.645344 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:21.645565 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.645573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.646699 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.646705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.648038 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.648051 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.648893 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.648900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.649038 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.649572 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:21.649591 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:21.652053 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:21.654875 (+ 2822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.654885 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.654891 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.655310 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.655635 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.655639 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.655643 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.655750 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.656243 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:21.656250 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:21.656403 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 20:47:21.656555 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 20:47:21.656847 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"thread_start_us":342,"threads_started":1}
I20260929 20:47:21.658922 7264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:47:21 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714841, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 20:47:21.670296 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.659515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42803 (local address 127.0.0.1:40552)
0929 20:47:21.659687 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:21.659722 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:21.659811 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:21.660399 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:21.660402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:21.660670 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:21.660810 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.660817 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.662007 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.662011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.662910 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.662920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.663596 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.663600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.663722 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.664181 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:21.664197 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:21.665991 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:21.668432 (+ 2441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.668436 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.668438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.668767 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.669044 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.669047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.669049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.669155 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.669577 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:21.669581 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:21.669703 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 20:47:21.669865 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 20:47:21.670078 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 20:47:21.670428 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.659702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40552 (local address 127.0.97.61:42803)
0929 20:47:21.660164 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:21.660168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:21.660183 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:47:21.660226 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:21.660230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:21.660283 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:21.660367 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:21.660935 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.661912 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.663089 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.663467 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.663556 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.666152 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:21.666156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:21.666159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:21.666189 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:21.668317 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.668848 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.668851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.668854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.668966 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.669225 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.669228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.669230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.669445 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:21.669552 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:21.670015 (+ 463us) server_negotiation.cc:299] Negotiation successful
0929 20:47:21.670174 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":245,"threads_started":1}
I20260929 20:47:21.671293 7264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:47:21 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714841, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 20:47:21.683175 7268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.671828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46467 (local address 127.0.0.1:38848)
0929 20:47:21.672003 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:21.672016 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:21.672085 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:21.672329 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:21.672331 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:21.672712 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:21.672869 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.672879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.673742 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.673747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.675055 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.675068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.675845 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.675849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.675965 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.676380 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:21.676394 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:21.678197 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:21.681082 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.681088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.681100 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.681417 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.681721 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.681726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.681730 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.681857 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.682283 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:21.682287 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:21.682407 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 20:47:21.682551 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 20:47:21.682782 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 20:47:21.683313 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.671880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38848 (local address 127.0.97.60:46467)
0929 20:47:21.672028 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:21.672032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:21.672049 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:47:21.672167 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:21.672170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:21.672217 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:21.672299 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:21.672986 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.673619 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.675188 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.675702 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.675791 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.678337 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:21.678341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:21.678344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:21.678368 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:21.680973 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.681515 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.681520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.681522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.681629 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.681947 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.681952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.681955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.682182 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:21.682296 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:21.682647 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 20:47:21.682927 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":59}
I20260929 20:47:21.687563 7264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1: Bootstrap starting.
I20260929 20:47:21.690449 7264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:21.691403 7264 log.cc:824] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:21.693831 7264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1: No bootstrap required, opened a new log
I20260929 20:47:21.701037 7264 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } }
I20260929 20:47:21.701335 7264 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:21.701426 7264 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee15b91d5caa47b6b8021324f9fc53b1, State: Initialized, Role: FOLLOWER
I20260929 20:47:21.701926 7264 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } }
I20260929 20:47:21.702629 7271 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } } }
I20260929 20:47:21.702901 7271 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:47:21.703532 7264 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 20:47:21.712652 7282 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:47:21.712778 7282 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:47:21.716120 7110 master_runner.cc:428] Detected that this master ee15b91d5caa47b6b8021324f9fc53b1 is joining an existing cluster
I20260929 20:47:21.716220 7110 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:46467
I20260929 20:47:21.737232 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.720013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46234 (local address 127.0.97.62:45167)
0929 20:47:21.720166 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:21.720172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:21.721046 (+ 874us) server_negotiation.cc:407] Connection header received
0929 20:47:21.721304 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:21.721309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:21.721386 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:21.721456 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:21.722905 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.724093 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.726178 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.726814 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.726905 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.730015 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:21.730022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:21.730026 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:21.730064 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:21.733315 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.734052 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.734057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.734060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.734214 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.735181 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.735187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.735190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.735532 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:21.735641 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:21.736429 (+ 788us) server_negotiation.cc:299] Negotiation successful
0929 20:47:21.736649 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 20:47:21.738720 7288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.720330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.0.1:46234)
0929 20:47:21.720956 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:21.720973 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:21.721147 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:21.721672 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:21.721677 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:21.722373 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:21.722739 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.722752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.724287 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.724293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.726015 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.726029 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.727214 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.727222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.727410 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.728073 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:21.728097 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:21.729809 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:21.733458 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.733463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.733468 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.733925 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.734625 (+ 700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.734630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.734634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.734818 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.735718 (+ 900us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:21.735722 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:21.736067 (+ 345us) client_negotiation.cc:769] Sending connection context
0929 20:47:21.737963 (+ 1896us) client_negotiation.cc:240] Negotiation successful
0929 20:47:21.738261 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"spinlock_wait_cycles":24448,"thread_start_us":347,"threads_started":1}
I20260929 20:47:21.742507 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.720116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38860 (local address 127.0.97.60:46467)
0929 20:47:21.720303 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:21.720312 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:21.728037 (+ 7725us) server_negotiation.cc:407] Connection header received
0929 20:47:21.728458 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:21.728463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:21.728532 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:21.728598 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:21.729979 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.730987 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.732804 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.733527 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.733611 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.736306 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:21.736312 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:21.736317 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:21.736348 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:21.739622 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.740445 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.740450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.740453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.740594 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.740987 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.740992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.740995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.741380 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:21.741510 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:21.741973 (+ 463us) server_negotiation.cc:299] Negotiation successful
0929 20:47:21.742164 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":34}
I20260929 20:47:21.743129 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.721184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.0.97.61:42803)
0929 20:47:21.721331 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:21.721336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:21.728206 (+ 6870us) server_negotiation.cc:407] Connection header received
0929 20:47:21.728931 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:21.728936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:21.729024 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:21.729089 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:21.730712 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.731696 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.733598 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.734161 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.734289 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.737016 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:21.737022 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:21.737026 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:21.737057 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:21.740054 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.740709 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.740713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.740716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.740876 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.741312 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.741316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.741320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.741620 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:21.741718 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:21.742604 (+ 886us) server_negotiation.cc:299] Negotiation successful
0929 20:47:21.742799 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:47:21.743417 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.720449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46467 (local address 127.0.0.1:38860)
0929 20:47:21.727948 (+ 7499us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:21.727965 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:21.728308 (+ 343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:21.728767 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:21.728771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:21.729514 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:21.729863 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.729873 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.731196 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.731200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.732665 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.732679 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.733853 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.733859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.734011 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.734597 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:21.734618 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:21.736129 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:21.739912 (+ 3783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.739917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.739921 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.740336 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.740718 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.740723 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.740726 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.740885 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.741591 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:21.741595 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:21.741747 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 20:47:21.742708 (+ 961us) client_negotiation.cc:240] Negotiation successful
0929 20:47:21.742944 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7378,"thread_start_us":7760,"threads_started":1}
I20260929 20:47:21.744534 7289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.721433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42803 (local address 127.0.0.1:40566)
0929 20:47:21.728118 (+ 6685us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:21.728135 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:21.728769 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:21.729330 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:21.729335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:21.730095 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:21.730530 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.730541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.731845 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.731850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.733454 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.733465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.734552 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.734559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.734711 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.735251 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:21.735281 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:21.736823 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:21.740181 (+ 3358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.740186 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.740190 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.740597 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.741006 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.741010 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.741012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.741176 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.741798 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:21.741802 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:21.741966 (+ 164us) client_negotiation.cc:769] Sending connection context
0929 20:47:21.743802 (+ 1836us) client_negotiation.cc:240] Negotiation successful
0929 20:47:21.744094 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6543,"mutex_wait_us":21,"thread_start_us":6615,"threads_started":1}
Sep 29 20:47:21 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714839, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 20:47:21.757488 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 6610
I20260929 20:47:21.757754 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.745625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46467 (local address 127.0.0.1:38864)
0929 20:47:21.746332 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:21.746349 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:21.746493 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:21.746838 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:21.746843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:21.747227 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:21.747382 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.747390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.748321 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.748327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.749143 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.749153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.749825 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.749831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.749983 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.750454 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:21.750469 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:21.752571 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:21.755334 (+ 2763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.755339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.755343 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.755852 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.756202 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.756207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.756211 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.756349 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.756916 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:21.756921 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:21.757035 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 20:47:21.757147 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 20:47:21.757399 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":371,"threads_started":1}
I20260929 20:47:21.758149 7267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.745763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.0.97.60:46467)
0929 20:47:21.745904 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:21.745909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:21.746410 (+ 501us) server_negotiation.cc:407] Connection header received
0929 20:47:21.746632 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:21.746635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:21.746714 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:21.746793 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:21.747509 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.748204 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.749274 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.749689 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.749798 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.752730 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:21.752735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:21.752738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:21.752763 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:21.755219 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.755969 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.755973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.755976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.756087 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.756461 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.756466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.756469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.756792 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:21.756932 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:21.757244 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 20:47:21.757546 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 20:47:21.759334 6549 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:46467: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 46467
}
attrs {
promote: true
}
}
dest_uuid: "1460986649014da8aff571f52348fcad"
cas_config_opid_index: 5
I20260929 20:47:21.760344 6549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: true } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: true } }
Sep 29 20:47:21 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260929 20:47:21.775520 7239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:21.779139 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.758440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48104 (local address 127.0.97.1:41329)
0929 20:47:21.759002 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:21.759011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:21.759060 (+ 49us) server_negotiation.cc:407] Connection header received
0929 20:47:21.759125 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:21.759134 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:21.759415 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:21.761134 (+ 1719us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:21.761369 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.762903 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.764795 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.766163 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.771215 (+ 5052us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.771474 (+ 259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:21.771485 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:21.771502 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:21.771534 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:21.775107 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.775954 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.775963 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.775966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.776103 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.776517 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.776521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.776525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.776979 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:21.778275 (+ 1296us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:21.778424 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 20:47:21.778726 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":261,"threads_started":1}
I20260929 20:47:21.779503 7239 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [term 1 FOLLOWER]: Refusing update from remote peer 1460986649014da8aff571f52348fcad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 20:47:21.780447 6733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:21.780786 7294 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ee15b91d5caa47b6b8021324f9fc53b1 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: true } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: true } } }
I20260929 20:47:21.781422 7295 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:47:21.784659 7294 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:46467
I20260929 20:47:21.785181 7110 master.cc:537] Master@127.0.97.60:46467 shutting down...
I20260929 20:47:21.785323 7293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1460986649014da8aff571f52348fcad" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: true } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: true } } }
I20260929 20:47:21.785650 7293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:21.789458 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.781450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42803 (local address 127.0.0.1:40576)
0929 20:47:21.782027 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:21.782048 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:21.782159 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:21.782706 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:21.782711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:21.782755 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:21.782934 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.782946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.784443 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.784449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.786877 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.786892 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.788709 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.788715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.788930 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.788985 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 20:47:21.789073 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 20:47:21.789210 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466}
I20260929 20:47:21.790069 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.781844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40576 (local address 127.0.97.61:42803)
0929 20:47:21.782053 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:21.782060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:21.782087 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:47:21.782298 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:21.782303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:21.782396 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:21.782512 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:21.783133 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.784245 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.787080 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.788245 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.788343 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.788572 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:21.789701 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0929 20:47:21.789832 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:47:21.791133 7158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 1 LEARNER]: Refusing update from remote peer 1460986649014da8aff571f52348fcad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 20:47:21.791771 7294 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 20:47:21.794826 7110 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:47:21.795182 7110 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:21.795362 7110 tablet_replica.cc:330] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1: stopping tablet replica
I20260929 20:47:21.801663 7110 master.cc:552] Master@127.0.97.60:46467 shutdown complete.
I20260929 20:47:21.801775 7110 master_runner.cc:305] Clearing existing system tablet
I20260929 20:47:21.802232 7110 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:21.802377 7110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:21.802259 7158 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ee15b91d5caa47b6b8021324f9fc53b1 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: true } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: true } } }
I20260929 20:47:21.809021 7110 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 20:47:21.809535 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.809062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46467 (local address 127.0.0.1:38870)
0929 20:47:21.809267 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:21.809370 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 20:47:21.812064 7186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1460986649014da8aff571f52348fcad. Latest consensus state: current_term: 1 leader_uuid: "1460986649014da8aff571f52348fcad" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: true } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: true } } }
I20260929 20:47:21.812376 7186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [sys.catalog]: This master's current role is: LEARNER
I20260929 20:47:21.812742 7301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1460986649014da8aff571f52348fcad" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: true } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: true } } }
I20260929 20:47:21.813097 7301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [sys.catalog]: This master's current role is: LEARNER
I20260929 20:47:21.814123 7296 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad: attempting to promote NON_VOTER 1097f58cae6a4662999174443b82b300 to VOTER
I20260929 20:47:21.814695 7303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:21.815095 7296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: true } }
I20260929 20:47:21.815374 7110 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:21.815615 7110 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:47:21.815728 7110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/wal
uuid: "ee15b91d5caa47b6b8021324f9fc53b1"
format_stamp: "Formatted at 2026-09-29 20:47:21 on dist-test-slave-blv7"
server_key: "95ca8ae0253b1d41cad10fdb14690203"
server_key_iv: "46f61bf642cef0e845f9dc7a28a73ccb"
server_key_version: "encryptionkey@0"
I20260929 20:47:21.819110 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.818674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46467 (local address 127.0.0.1:38882)
0929 20:47:21.818835 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:21.818931 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 20:47:21.820282 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.819943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46467 (local address 127.0.0.1:38894)
0929 20:47:21.820065 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:21.820145 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 20:47:21.820586 6536 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad -> Peer ee15b91d5caa47b6b8021324f9fc53b1 (127.0.97.60:46467): Couldn't send request to peer ee15b91d5caa47b6b8021324f9fc53b1. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46467: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 20:47:21.821141 7110 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:21.821671 7158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 1 LEARNER]: Refusing update from remote peer 1460986649014da8aff571f52348fcad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 20:47:21.822175 7293 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 20:47:21.824295 7296 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1097f58cae6a4662999174443b82b300 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: true } } }
I20260929 20:47:21.828215 7110 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 20:47:21.828356 7110 log.cc:1196] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 20:47:21.828799 7110 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1: Deleting consensus metadata
I20260929 20:47:21.829017 7110 master_runner.cc:315] Copying system tablet from 127.0.97.62:45167
I20260929 20:47:21.829229 7158 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1097f58cae6a4662999174443b82b300 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: true } } }
I20260929 20:47:21.829533 7296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1460986649014da8aff571f52348fcad" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: true } } }
I20260929 20:47:21.829691 7295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1460986649014da8aff571f52348fcad" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: true } } }
I20260929 20:47:21.830125 7296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:21.830150 7295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:21.832429 7301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1460986649014da8aff571f52348fcad" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: true } } }
I20260929 20:47:21.832604 7301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:47:21.836375 7110 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:45167
I20260929 20:47:21.851620 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.837130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46242 (local address 127.0.97.62:45167)
0929 20:47:21.837325 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:21.837331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:21.837803 (+ 472us) server_negotiation.cc:407] Connection header received
0929 20:47:21.837958 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:21.837963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:21.838043 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:21.838144 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:21.839099 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.840291 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.842148 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.842779 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.842869 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.845709 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:21.845716 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:21.845721 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:21.845757 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:21.848949 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.849656 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.849661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.849665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.849800 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:21.850176 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:21.850180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:21.850184 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:21.850477 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:21.850629 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:21.851085 (+ 456us) server_negotiation.cc:299] Negotiation successful
0929 20:47:21.851282 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 20:47:21.851928 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:21.837111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.0.1:46242)
0929 20:47:21.837691 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:21.837709 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:21.837848 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:21.838197 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:21.838201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:21.838708 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:21.838942 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.838954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.840487 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.840493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.841989 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:21.842005 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:21.843152 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:21.843158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:21.843338 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:21.844457 (+ 1119us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:21.844484 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:21.845513 (+ 1029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:21.849077 (+ 3564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.849082 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.849086 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.849537 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.849908 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:21.849913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:21.849917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:21.850079 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:21.850592 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:21.850597 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:21.850783 (+ 186us) client_negotiation.cc:769] Sending connection context
0929 20:47:21.850932 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 20:47:21.851256 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":338,"threads_started":1}
I20260929 20:47:21.852463 6569 tablet_copy_service.cc:140] P 1460986649014da8aff571f52348fcad: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ee15b91d5caa47b6b8021324f9fc53b1 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:46242)
I20260929 20:47:21.852617 6569 tablet_copy_service.cc:161] P 1460986649014da8aff571f52348fcad: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ee15b91d5caa47b6b8021324f9fc53b1 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:46242: session id = ee15b91d5caa47b6b8021324f9fc53b1-00000000000000000000000000000000
I20260929 20:47:21.856176 6569 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 20:47:21.857846 7110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 20:47:21.864765 7110 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1: tablet copy: Starting download of 0 data blocks...
I20260929 20:47:21.865017 7110 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1: tablet copy: Starting download of 1 WAL segments...
I20260929 20:47:21.867250 7110 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 20:47:21.871279 6569 tablet_copy_service.cc:342] P 1460986649014da8aff571f52348fcad: Request end of tablet copy session ee15b91d5caa47b6b8021324f9fc53b1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:46242
I20260929 20:47:21.871405 6569 tablet_copy_service.cc:433] P 1460986649014da8aff571f52348fcad: ending tablet copy session ee15b91d5caa47b6b8021324f9fc53b1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ee15b91d5caa47b6b8021324f9fc53b1
I20260929 20:47:21.884724 7110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:21.887357 7310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:21.887795 7311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:21.895364 7110 server_base.cc:1034] running on GCE node
W20260929 20:47:21.895617 7313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:21.896018 7110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:21.896623 7110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:21.897817 7110 hybrid_clock.cc:648] HybridClock initialized: now 1790714841897776 us; error 32 us; skew 500 ppm
I20260929 20:47:21.898044 7110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:21 dist-test-slave-blv7 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714841, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:21.900152 7110 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260929 20:47:21.901708 7110 webserver.cc:460] Webserver started at http://127.0.97.60:39443/ using document root <none> and password file <none>
I20260929 20:47:21.901994 7110 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:21.902081 7110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:21.904934 7110 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 20:47:21.907105 7320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:21.907513 7110 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 20:47:21.907672 7110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/wal
uuid: "ee15b91d5caa47b6b8021324f9fc53b1"
format_stamp: "Formatted at 2026-09-29 20:47:21 on dist-test-slave-blv7"
server_key: "95ca8ae0253b1d41cad10fdb14690203"
server_key_iv: "46f61bf642cef0e845f9dc7a28a73ccb"
server_key_version: "encryptionkey@0"
I20260929 20:47:21.907898 7110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:21.923573 7110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:21.927465 7110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:21.943966 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 6610
I20260929 20:47:21.966778 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41329
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45167
--webserver_port=46629
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:45167,127.0.97.61:42803
--tserver_master_addrs=127.0.97.62:45167,127.0.97.61:42803,127.0.97.60:46467 with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:47:21.972774 7110 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:46467
I20260929 20:47:21.972807 7371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:46467 every 8 connection(s)
I20260929 20:47:21.973691 7110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/master-2/data/info.pb
I20260929 20:47:21.978706 7372 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:47:21.982419 7372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1: Bootstrap starting.
I20260929 20:47:21.999083 7372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:47:21.999518 7372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1: Bootstrap complete.
I20260929 20:47:22.000473 7372 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: true } }
I20260929 20:47:22.000813 7372 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee15b91d5caa47b6b8021324f9fc53b1, State: Initialized, Role: LEARNER
I20260929 20:47:22.001083 7372 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: true } }
I20260929 20:47:22.002278 7372 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:47:22.002086 7376 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: true } } }
I20260929 20:47:22.002636 7376 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [sys.catalog]: This master's current role is: LEARNER
I20260929 20:47:22.013806 7387 catalog_manager.cc:1259] Loaded cluster ID: 48312dabe6844bb0a0147505a1576ca1
I20260929 20:47:22.013962 7387 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1: loading cluster ID for follower catalog manager: success
I20260929 20:47:22.017325 7387 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1: acquiring CA information for follower catalog manager: success
I20260929 20:47:22.020008 7387 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 20:47:22.283702 7374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:22.283946 7374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:22.284072 7374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:22.291623 7374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:22.291714 7374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:22.291857 7374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:47:22.301738 7374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=46629
--tserver_master_addrs=127.0.97.62:45167,127.0.97.61:42803,127.0.97.60:46467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:22.303009 7374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:22.304343 7374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:22.316845 7392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:22.316916 7395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:22.316845 7393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:22.317648 7374 server_base.cc:1034] running on GCE node
I20260929 20:47:22.318187 7374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:22.319046 7374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:22.320262 7374 hybrid_clock.cc:648] HybridClock initialized: now 1790714842320209 us; error 45 us; skew 500 ppm
I20260929 20:47:22.320658 7374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:22 dist-test-slave-blv7 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714842, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:22.323439 7374 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 20:47:22.324786 7374 webserver.cc:460] Webserver started at http://127.0.97.1:46629/ using document root <none> and password file <none>
I20260929 20:47:22.325385 7374 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:22.325505 7374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:22.330615 7374 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:47:22.333734 7402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:22.335007 7374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:47:22.335220 7374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "d343b75e87f142f29eb74970160c325e"
format_stamp: "Formatted at 2026-09-29 20:47:19 on dist-test-slave-blv7"
server_key: "b9f713e1aa084cf26417e50b85c34946"
server_key_iv: "23a7426c929c8c94891165d7fe00ed4f"
server_key_version: "encryptionkey@0"
I20260929 20:47:22.336222 7374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:22.366374 7374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:22.370325 7374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:22.370579 7374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:22.371945 7374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:22.373210 7374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:22.373301 7374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:22.373402 7374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:22.373471 7374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:22.405557 7374 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41329
I20260929 20:47:22.405614 7514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41329 every 8 connection(s)
I20260929 20:47:22.406888 7374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:47:22.409569 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 7374
I20260929 20:47:22.409888 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 6744
Sep 29 20:47:22 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714842, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 20:47:22.441182 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:22.412248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46467 (local address 127.0.0.1:38900)
0929 20:47:22.425475 (+ 13227us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:22.425493 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:22.425611 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:22.426595 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:22.426600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:22.426628 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:22.426835 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:22.426846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:22.430376 (+ 3530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:22.430382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:22.432095 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:22.432109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:22.438504 (+ 6395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:22.438510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:22.438688 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:22.440582 (+ 1894us) client_negotiation.cc:769] Sending connection context
0929 20:47:22.440666 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 20:47:22.440849 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13123,"mutex_wait_us":1824,"thread_start_us":13047,"threads_started":1}
Sep 29 20:47:22 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714842, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 20:47:22.441939 7526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:22.412643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38900 (local address 127.0.97.60:46467)
0929 20:47:22.425190 (+ 12547us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:22.425196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:22.425720 (+ 524us) server_negotiation.cc:407] Connection header received
0929 20:47:22.425781 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:22.425786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:22.425856 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:22.425958 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:22.429093 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:22.430196 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:22.434975 (+ 4779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:22.436132 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:22.436273 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:22.436512 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:22.441493 (+ 4981us) server_negotiation.cc:299] Negotiation successful
0929 20:47:22.441625 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12444,"thread_start_us":12338,"threads_started":1}
I20260929 20:47:22.442725 7347 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 20:47:22.444053 7376 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1460986649014da8aff571f52348fcad. Latest consensus state: current_term: 1 leader_uuid: "1460986649014da8aff571f52348fcad" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: true } } }
I20260929 20:47:22.444311 7376 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [sys.catalog]: This master's current role is: LEARNER
I20260929 20:47:22.447772 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:22.410586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45943 (local address 127.0.97.60:46467)
0929 20:47:22.423813 (+ 13227us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:22.423817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:22.424244 (+ 427us) server_negotiation.cc:407] Connection header received
0929 20:47:22.424366 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:22.424371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:22.424447 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:22.424544 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:22.427046 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:22.428437 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:22.430796 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:22.431421 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:22.431508 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:22.436050 (+ 4542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:22.436056 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:22.436061 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:22.436098 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:22.439371 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:22.441016 (+ 1645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:22.441021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:22.441024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:22.441157 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:22.442849 (+ 1692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:22.442854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:22.442857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:22.443651 (+ 794us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:22.443780 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:22.447215 (+ 3435us) server_negotiation.cc:299] Negotiation successful
0929 20:47:22.447421 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13124,"thread_start_us":13017,"threads_started":1}
I20260929 20:47:22.450179 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:22.410689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46467 (local address 127.0.97.1:45943)
0929 20:47:22.424136 (+ 13447us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:22.424150 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:22.424256 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:22.425014 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:22.425026 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:22.425996 (+ 970us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:22.426859 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:22.426881 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:22.428615 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:22.428621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:22.430651 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:22.430664 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:22.431827 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:22.431834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:22.432069 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:22.432815 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:22.432844 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:22.435823 (+ 2979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:22.439547 (+ 3724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:22.439555 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:22.439573 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:22.440888 (+ 1315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:22.441501 (+ 613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:22.441505 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:22.441508 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:22.442310 (+ 802us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:22.444473 (+ 2163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:22.444482 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:22.444881 (+ 399us) client_negotiation.cc:769] Sending connection context
0929 20:47:22.448217 (+ 3336us) client_negotiation.cc:240] Negotiation successful
0929 20:47:22.448754 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13360,"thread_start_us":27281,"threads_started":1}
I20260929 20:47:22.451440 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:22.410113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42803 (local address 127.0.97.1:53995)
0929 20:47:22.419681 (+ 9568us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:22.419748 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:22.423137 (+ 3389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:22.427053 (+ 3916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:22.427057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:22.427437 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:22.427676 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:22.427686 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:22.432275 (+ 4589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:22.432280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:22.433687 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:22.433712 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:22.439005 (+ 5293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:22.439011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:22.439257 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:22.439790 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:22.439812 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:22.442300 (+ 2488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:22.448591 (+ 6291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:22.448607 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:22.448610 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:22.449142 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:22.449572 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:22.449576 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:22.449580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:22.449763 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:22.450387 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:22.450393 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:22.450524 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 20:47:22.450653 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 20:47:22.450934 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9169,"thread_start_us":8971,"threads_started":1}
I20260929 20:47:22.452482 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:22.411854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53995 (local address 127.0.97.61:42803)
0929 20:47:22.424648 (+ 12794us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:22.424653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:22.424670 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:47:22.424747 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:22.424752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:22.424818 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:22.427832 (+ 3014us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:22.428006 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:22.428936 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:22.433876 (+ 4940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:22.434837 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:22.439558 (+ 4721us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:22.442569 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:22.442575 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:22.442580 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:22.442624 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:22.448423 (+ 5799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:22.449296 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:22.449302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:22.449305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:22.449441 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:22.449934 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:22.449939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:22.449942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:22.450262 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:22.451849 (+ 1587us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:22.451984 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 20:47:22.452188 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12678,"thread_start_us":12600,"threads_started":1}
Sep 29 20:47:22 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Sep 29 20:47:22 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714842, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:47:22.453815 7517 heartbeater.cc:347] Connected to a master server at 127.0.97.61:42803
I20260929 20:47:22.454118 7515 heartbeater.cc:347] Connected to a master server at 127.0.97.60:46467
I20260929 20:47:22.454175 7517 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:22.454334 7515 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:22.455634 7517 heartbeater.cc:511] Master 127.0.97.61:42803 requested a full tablet report, sending...
I20260929 20:47:22.455655 7515 heartbeater.cc:511] Master 127.0.97.60:46467 requested a full tablet report, sending...
I20260929 20:47:22.457893 7148 ts_manager.cc:194] Registered new tserver with Master: d343b75e87f142f29eb74970160c325e (127.0.97.1:41329)
I20260929 20:47:22.457893 7337 ts_manager.cc:194] Registered new tserver with Master: d343b75e87f142f29eb74970160c325e (127.0.97.1:41329)
I20260929 20:47:22.461076 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:22.420364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.97.1:47845)
0929 20:47:22.438207 (+ 17843us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:22.438225 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:22.438323 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:22.438743 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:22.438747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:22.439364 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:22.439946 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:22.439962 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:22.443887 (+ 3925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:22.443892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:22.446239 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:22.446254 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:22.450953 (+ 4699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:22.450959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:22.451197 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:22.451795 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:22.451816 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:22.454556 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:22.458618 (+ 4062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:22.458623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:22.458628 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:22.459089 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:22.459454 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:22.459458 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:22.459461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:22.459613 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:22.460157 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:22.460162 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:22.460284 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 20:47:22.460406 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 20:47:22.460675 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17731,"thread_start_us":3582,"threads_started":1}
I20260929 20:47:22.461097 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:22.411212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33664 (local address 127.0.97.2:33431)
0929 20:47:22.426507 (+ 15295us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:22.426516 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:22.437198 (+ 10682us) server_negotiation.cc:407] Connection header received
0929 20:47:22.437274 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:22.437282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:22.437535 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:22.437677 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:22.440600 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:22.445467 (+ 4867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:22.447387 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:22.448258 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:22.448379 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:22.454043 (+ 5664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:22.454058 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:22.454074 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:22.454124 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:22.458035 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:22.458886 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:22.458894 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:22.458898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:22.459073 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:22.459459 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:22.459463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:22.459466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:22.459864 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:22.460067 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:22.460477 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 20:47:22.460718 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15141,"thread_start_us":15016,"threads_started":1}
I20260929 20:47:22.461133 7528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:22.425526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47845 (local address 127.0.97.62:45167)
0929 20:47:22.438474 (+ 12948us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:22.438478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:22.438500 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:47:22.438538 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:22.438541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:22.438603 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:22.438695 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:22.440075 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:22.442691 (+ 2616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:22.446401 (+ 3710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:22.450562 (+ 4161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:22.450700 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:22.454797 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:22.454803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:22.454808 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:22.454848 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:22.458475 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:22.459203 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:22.459208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:22.459211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:22.459345 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:22.459728 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:22.459732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:22.459734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:22.460022 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:22.460220 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:22.460524 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 20:47:22.460730 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12841,"thread_start_us":12756,"threads_started":1}
I20260929 20:47:22.461786 7516 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45167
I20260929 20:47:22.461908 7516 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:22.462280 6867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:22.462340 7516 heartbeater.cc:511] Master 127.0.97.62:45167 requested a full tablet report, sending...
I20260929 20:47:22.463514 6549 ts_manager.cc:194] Re-registered known tserver with Master: d343b75e87f142f29eb74970160c325e (127.0.97.1:41329)
I20260929 20:47:22.464552 6549 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47845
I20260929 20:47:22.543282 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 6744
I20260929 20:47:22.555676 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33431
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45167
--webserver_port=32963
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:45167,127.0.97.61:42803
--tserver_master_addrs=127.0.97.62:45167,127.0.97.61:42803,127.0.97.60:46467 with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:22.865911 7530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:22.866170 7530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:22.866315 7530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 20:47:22.872087 7294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1460986649014da8aff571f52348fcad" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: true } } }
I20260929 20:47:22.872434 7294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:22.873577 7535 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad: attempting to promote NON_VOTER ee15b91d5caa47b6b8021324f9fc53b1 to VOTER
W20260929 20:47:22.873978 7530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:22.874090 7530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:22.874289 7530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:47:22.874264 7535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: false } }
I20260929 20:47:22.876348 7347 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [term 1 LEARNER]: Refusing update from remote peer 1460986649014da8aff571f52348fcad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 20:47:22.876456 7158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 1 FOLLOWER]: Refusing update from remote peer 1460986649014da8aff571f52348fcad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 20:47:22.876878 7535 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 20:47:22.877218 7294 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 20:47:22.879892 7294 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ee15b91d5caa47b6b8021324f9fc53b1 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: false } } }
I20260929 20:47:22.880446 7158 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ee15b91d5caa47b6b8021324f9fc53b1 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: false } } }
I20260929 20:47:22.882546 7294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1460986649014da8aff571f52348fcad" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: false } } }
I20260929 20:47:22.882733 7294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:22.882942 7534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1460986649014da8aff571f52348fcad" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: false } } }
I20260929 20:47:22.883236 7534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:22.887072 7540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1460986649014da8aff571f52348fcad" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: false } } }
I20260929 20:47:22.887423 7540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:47:22.887446 7534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1460986649014da8aff571f52348fcad" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: false } } }
I20260929 20:47:22.887670 7534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:22.888619 7530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=32963
--tserver_master_addrs=127.0.97.62:45167,127.0.97.61:42803,127.0.97.60:46467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:22.888294 7347 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ee15b91d5caa47b6b8021324f9fc53b1 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: false } } }
I20260929 20:47:22.889856 7530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:22.891063 7376 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1460986649014da8aff571f52348fcad" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: false } } }
I20260929 20:47:22.891273 7376 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:47:22.891710 7530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:22.904183 7544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:22.904215 7543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:22.904242 7546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:22.904796 7530 server_base.cc:1034] running on GCE node
I20260929 20:47:22.905315 7530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:22.906430 7530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:22.907637 7530 hybrid_clock.cc:648] HybridClock initialized: now 1790714842907587 us; error 49 us; skew 500 ppm
I20260929 20:47:22.907953 7530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:22 dist-test-slave-blv7 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714842, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:22.910915 7530 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 20:47:22.912364 7530 webserver.cc:460] Webserver started at http://127.0.97.2:32963/ using document root <none> and password file <none>
I20260929 20:47:22.913024 7530 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:22.913137 7530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:22.918201 7530 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 20:47:22.920992 7553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:22.922435 7530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:47:22.922569 7530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "16a15205be8e418ba7b0856023a000ee"
format_stamp: "Formatted at 2026-09-29 20:47:20 on dist-test-slave-blv7"
server_key: "4c78c13440cbc1cc5d34f09eca573900"
server_key_iv: "994a7d49a06ed5ddf2d8a7106a2daf30"
server_key_version: "encryptionkey@0"
I20260929 20:47:22.923584 7530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:22.939149 7530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:22.943001 7530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:22.943261 7530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:22.944087 7530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:22.945305 7530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:22.945387 7530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:22.945461 7530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:22.945514 7530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:22.974545 7530 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33431
I20260929 20:47:22.974594 7665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33431 every 8 connection(s)
I20260929 20:47:22.975907 7530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:47:22.982564 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 7530
I20260929 20:47:22.982848 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 6877
Sep 29 20:47:23 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714842, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 29 20:47:23 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Sep 29 20:47:23 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714842, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 20:47:23 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714842, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 20:47:23.023243 7671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:22.979679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57069 (local address 127.0.97.62:45167)
0929 20:47:23.001592 (+ 21913us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:23.001598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:23.001641 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:47:23.002729 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:23.002736 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:23.002833 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:23.005322 (+ 2489us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:23.005488 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.006615 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.009585 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.010188 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.010314 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.014952 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:23.014958 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:23.014963 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:23.015000 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:23.018435 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.019463 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.019469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.019472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.019615 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.020182 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.020187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.020190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.020560 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:23.020695 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:23.022651 (+ 1956us) server_negotiation.cc:299] Negotiation successful
0929 20:47:23.022863 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21801,"thread_start_us":21687,"threads_started":1}
I20260929 20:47:23.023473 7670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:22.979129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.97.2:57069)
0929 20:47:22.997436 (+ 18307us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:22.997487 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:23.000028 (+ 2541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:23.003401 (+ 3373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:23.003415 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:23.004238 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:23.005129 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:23.005155 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.006807 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.006813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:23.009439 (+ 2626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:23.009450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.010578 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.010585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:23.010849 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.011624 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:23.011653 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:23.014701 (+ 3048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:23.018579 (+ 3878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:23.018587 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:23.018604 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:23.019318 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:23.019748 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:23.019752 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:23.019756 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:23.019990 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:23.020762 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:23.020793 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:23.021258 (+ 465us) client_negotiation.cc:769] Sending connection context
0929 20:47:23.021545 (+ 287us) client_negotiation.cc:240] Negotiation successful
0929 20:47:23.022054 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17999,"thread_start_us":17778,"threads_started":1}
I20260929 20:47:23.024911 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:22.983848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40362 (local address 127.0.97.3:39649)
0929 20:47:23.001953 (+ 18105us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:23.001963 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:23.004100 (+ 2137us) server_negotiation.cc:407] Connection header received
0929 20:47:23.004182 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:23.004193 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:23.004508 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:23.004669 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:23.006976 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.008695 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.010598 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.011486 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.011614 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.017801 (+ 6187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:23.017813 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:23.017830 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:23.017867 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:23.021864 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.022527 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.022536 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.022539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.022677 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.023127 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.023132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.023135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.023516 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:23.023774 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:23.024338 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 20:47:23.024561 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17649,"thread_start_us":17767,"threads_started":1}
I20260929 20:47:23.024969 7667 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45167
I20260929 20:47:23.025243 7667 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:23.025938 7667 heartbeater.cc:511] Master 127.0.97.62:45167 requested a full tablet report, sending...
I20260929 20:47:23.026235 7000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:23.027825 6549 ts_manager.cc:194] Re-registered known tserver with Master: 16a15205be8e418ba7b0856023a000ee (127.0.97.2:33431)
I20260929 20:47:23.028914 6549 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57069
I20260929 20:47:23.029668 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:22.979703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46467 (local address 127.0.97.2:35659)
0929 20:47:23.003254 (+ 23551us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:23.003272 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:23.003386 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:23.009723 (+ 6337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:23.009728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:23.010236 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:23.010493 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:23.010502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.012411 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.012417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:23.013873 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:23.013886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.014956 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.014962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:23.015138 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.015881 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:23.015901 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:23.018253 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:23.026615 (+ 8362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:23.026619 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:23.026622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:23.027182 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:23.027554 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:23.027558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:23.027561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:23.027701 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:23.028555 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:23.028561 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:23.028709 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 20:47:23.028834 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 20:47:23.029224 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23428,"mutex_wait_us":187,"thread_start_us":28263,"threads_started":1}
I20260929 20:47:23.030675 7669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:22.978592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35659 (local address 127.0.97.60:46467)
0929 20:47:22.990988 (+ 12396us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:22.990995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:23.009301 (+ 18306us) server_negotiation.cc:407] Connection header received
0929 20:47:23.009361 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:23.009367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:23.009446 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:23.009535 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:23.010658 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.012265 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.014053 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.014635 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.014748 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.018453 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:23.018459 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:23.018463 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:23.018498 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:23.024639 (+ 6141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.027314 (+ 2675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.027319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.027323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.027457 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.027865 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.027870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.027874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.028342 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:23.028483 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:23.030083 (+ 1600us) server_negotiation.cc:299] Negotiation successful
0929 20:47:23.030238 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12255,"thread_start_us":12394,"threads_started":1}
I20260929 20:47:23.031344 7666 heartbeater.cc:347] Connected to a master server at 127.0.97.60:46467
I20260929 20:47:23.031463 7666 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:23.032024 7666 heartbeater.cc:511] Master 127.0.97.60:46467 requested a full tablet report, sending...
I20260929 20:47:23.032805 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:22.998219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42803 (local address 127.0.97.2:34111)
0929 20:47:23.009047 (+ 10828us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:23.009066 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:23.009198 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:23.014357 (+ 5159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:23.014363 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:23.014992 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:23.015443 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:23.015454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.016858 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.016863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:23.018327 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:23.018341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.019237 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.019243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:23.019482 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.019945 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:23.019965 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:23.022459 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:23.029892 (+ 7433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:23.029898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:23.029902 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:23.030718 (+ 816us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:23.031151 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:23.031156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:23.031159 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:23.031312 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:23.031899 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:23.031904 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:23.032071 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 20:47:23.032227 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 20:47:23.032475 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10677,"mutex_wait_us":72,"thread_start_us":4778,"threads_started":1}
I20260929 20:47:23.032809 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:22.998296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34111 (local address 127.0.97.61:42803)
0929 20:47:23.013936 (+ 15640us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:23.013943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:23.013963 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:47:23.014019 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:23.014024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:23.014093 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:23.015221 (+ 1128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:23.015657 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.016705 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.018516 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.019042 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.019729 (+ 687us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.022661 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:23.022666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:23.022670 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:23.022715 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:23.028766 (+ 6051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.030866 (+ 2100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.030871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.030875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.031008 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.031447 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.031451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.031455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.031785 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:23.031908 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:23.032313 (+ 405us) server_negotiation.cc:299] Negotiation successful
0929 20:47:23.032519 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15504,"thread_start_us":15900,"threads_started":1}
I20260929 20:47:23.033128 7337 ts_manager.cc:194] Registered new tserver with Master: 16a15205be8e418ba7b0856023a000ee (127.0.97.2:33431)
I20260929 20:47:23.033579 7668 heartbeater.cc:347] Connected to a master server at 127.0.97.61:42803
I20260929 20:47:23.033681 7668 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:23.033936 7668 heartbeater.cc:511] Master 127.0.97.61:42803 requested a full tablet report, sending...
I20260929 20:47:23.034775 7148 ts_manager.cc:194] Registered new tserver with Master: 16a15205be8e418ba7b0856023a000ee (127.0.97.2:33431)
I20260929 20:47:23.110627 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 6877
I20260929 20:47:23.123967 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39649
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45167
--webserver_port=45121
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:45167,127.0.97.61:42803
--tserver_master_addrs=127.0.97.62:45167,127.0.97.61:42803,127.0.97.60:46467 with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:23.448027 7679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:23.448339 7679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:23.448473 7679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:23.456099 7679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:23.456228 7679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:23.456372 7679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:47:23.466531 7679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39649
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45121
--tserver_master_addrs=127.0.97.62:45167,127.0.97.61:42803,127.0.97.60:46467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:23.466955 7516 heartbeater.cc:503] Master 127.0.97.62:45167 was elected leader, sending a full tablet report...
I20260929 20:47:23.468078 7679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:23.469380 7679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:23.481618 7684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:23.481753 7685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:23.481853 7687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:23.482990 7679 server_base.cc:1034] running on GCE node
I20260929 20:47:23.483578 7679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:23.484450 7679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:23.485656 7679 hybrid_clock.cc:648] HybridClock initialized: now 1790714843485598 us; error 52 us; skew 500 ppm
I20260929 20:47:23.485949 7679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:23 dist-test-slave-blv7 krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714843, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:23.488804 7679 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 20:47:23.490084 7679 webserver.cc:460] Webserver started at http://127.0.97.3:45121/ using document root <none> and password file <none>
I20260929 20:47:23.490712 7679 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:23.490811 7679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:23.495777 7679 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:47:23.498198 7694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:23.499475 7679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:47:23.499614 7679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "dd36b520f02e4c7fa3082e278f14b414"
format_stamp: "Formatted at 2026-09-29 20:47:20 on dist-test-slave-blv7"
server_key: "f5c371d0477c4cb20806a21e32e50692"
server_key_iv: "c6b125ad31a9740668f7721744108457"
server_key_version: "encryptionkey@0"
I20260929 20:47:23.500527 7679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:23.528159 7679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:23.532089 7679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:23.532335 7679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:23.533294 7679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:23.534578 7679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:23.534659 7679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:23.534763 7679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:23.534828 7679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:23.562368 7679 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39649
I20260929 20:47:23.562429 7806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39649 every 8 connection(s)
I20260929 20:47:23.563733 7679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:47:23.572017 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 7679
Sep 29 20:47:23 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714843, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 29 20:47:23 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714843, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 20:47:23 dist-test-slave-blv7 krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714843, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:47:23.611907 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:23.567110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42803 (local address 127.0.97.3:33045)
0929 20:47:23.578152 (+ 11042us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:23.578178 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:23.578315 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:23.583774 (+ 5459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:23.583789 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:23.584776 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:23.585689 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:23.585716 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.587467 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.587473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:23.589525 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:23.589540 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.590942 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.590949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:23.591273 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.592005 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:23.592035 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:23.596015 (+ 3980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:23.601801 (+ 5786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:23.601809 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:23.601826 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:23.605422 (+ 3596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:23.606976 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:23.606981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:23.606985 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:23.607250 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:23.608727 (+ 1477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:23.608736 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:23.609276 (+ 540us) client_negotiation.cc:769] Sending connection context
0929 20:47:23.609653 (+ 377us) client_negotiation.cc:240] Negotiation successful
0929 20:47:23.610170 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10874,"mutex_wait_us":342,"thread_start_us":10889,"threads_started":1}
I20260929 20:47:23.611934 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:23.566554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33045 (local address 127.0.97.61:42803)
0929 20:47:23.583121 (+ 16567us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:23.583128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:23.583149 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:47:23.583214 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:23.583221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:23.583298 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:23.585923 (+ 2625us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:23.586076 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.587253 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.589813 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.590558 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.590667 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.596247 (+ 5580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:23.596253 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:23.596257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:23.596293 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:23.601652 (+ 5359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.606683 (+ 5031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.606688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.606691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.606854 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.607856 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.607861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.607865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.608509 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:23.608640 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:23.611381 (+ 2741us) server_negotiation.cc:299] Negotiation successful
0929 20:47:23.611583 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16416,"thread_start_us":16293,"threads_started":1}
I20260929 20:47:23.614188 7809 heartbeater.cc:347] Connected to a master server at 127.0.97.61:42803
I20260929 20:47:23.614497 7809 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:23.615383 7809 heartbeater.cc:511] Master 127.0.97.61:42803 requested a full tablet report, sending...
I20260929 20:47:23.617103 7148 ts_manager.cc:194] Registered new tserver with Master: dd36b520f02e4c7fa3082e278f14b414 (127.0.97.3:39649)
I20260929 20:47:23.621223 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:23.578974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51717 (local address 127.0.97.62:45167)
0929 20:47:23.594573 (+ 15599us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:23.594578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:23.594596 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:47:23.594663 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:23.594667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:23.594740 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:23.594849 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:23.595864 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.596900 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.598670 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.599271 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.599378 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.612206 (+ 12828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:23.612211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:23.612215 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:23.612251 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:23.617033 (+ 4782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.617874 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.617879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.617882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.618018 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.618431 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.618435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.618438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.618745 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:23.618856 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:23.620369 (+ 1513us) server_negotiation.cc:299] Negotiation successful
0929 20:47:23.620572 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15488,"spinlock_wait_cycles":2816,"thread_start_us":16000,"threads_started":1}
I20260929 20:47:23.621816 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:23.567110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46467 (local address 127.0.97.3:55027)
0929 20:47:23.568107 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:23.568161 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:23.570811 (+ 2650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:23.595661 (+ 24850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:23.595670 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:23.596118 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:23.596352 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:23.596363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.597788 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.597794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:23.599953 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:23.599966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.600910 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.600917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:23.601075 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.601653 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:23.601675 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:23.604400 (+ 2725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:23.616624 (+ 12224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:23.616629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:23.616634 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:23.617048 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:23.619580 (+ 2532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:23.619585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:23.619588 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:23.619789 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:23.620424 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:23.620429 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:23.620557 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 20:47:23.620671 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 20:47:23.620997 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":41,"thread_start_us":469,"threads_started":1}
I20260929 20:47:23.621953 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:23.600968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38912 (local address 127.0.97.60:46467)
0929 20:47:23.603969 (+ 3001us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:23.603974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:23.603998 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:47:23.604040 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:23.604044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:23.604114 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:23.604220 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:23.606566 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.607709 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.609935 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.610557 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.610671 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.614436 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:23.614444 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:23.614449 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:23.614485 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:23.618499 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.619290 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.619295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.619299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.619460 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.619903 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.619907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.619911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.620297 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:23.620511 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:23.621505 (+ 994us) server_negotiation.cc:299] Negotiation successful
0929 20:47:23.621647 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":2884,"spinlock_wait_cycles":6784,"thread_start_us":2782,"threads_started":1}
I20260929 20:47:23.623200 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:23.566540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55027 (local address 127.0.97.60:46467)
0929 20:47:23.595357 (+ 28817us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:23.595362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:23.595380 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:47:23.595432 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:23.595436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:23.595503 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:23.595628 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:23.596523 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.597605 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.600073 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.600691 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.601449 (+ 758us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.604567 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:23.604572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:23.604577 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:23.604612 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:23.615685 (+ 11073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.619327 (+ 3642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.619331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.619334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.619460 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.619910 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.619915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.619918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.620297 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:23.622412 (+ 2115us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:23.622532 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 20:47:23.622740 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":28702,"spinlock_wait_cycles":10752,"thread_start_us":29172,"threads_started":1}
I20260929 20:47:23.623857 7807 heartbeater.cc:347] Connected to a master server at 127.0.97.60:46467
I20260929 20:47:23.623991 7807 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:23.624439 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:23.578897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.97.3:51717)
0929 20:47:23.582398 (+ 3501us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:23.582421 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:23.582536 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:23.594864 (+ 12328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:23.594869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:23.595391 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:23.595692 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:23.595711 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.597027 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.597033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:23.598534 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:23.598544 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.599630 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.599638 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:23.600138 (+ 500us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.601219 (+ 1081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:23.601245 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:23.608188 (+ 6943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:23.617260 (+ 9072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:23.617264 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:23.617267 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:23.617752 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:23.618178 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:23.618183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:23.618186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:23.618315 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:23.618943 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:23.618948 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:23.620114 (+ 1166us) client_negotiation.cc:769] Sending connection context
0929 20:47:23.621026 (+ 912us) client_negotiation.cc:240] Negotiation successful
0929 20:47:23.623974 (+ 2948us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3382,"mutex_wait_us":3,"thread_start_us":3720,"threads_started":1}
I20260929 20:47:23.624483 7807 heartbeater.cc:511] Master 127.0.97.60:46467 requested a full tablet report, sending...
I20260929 20:47:23.625634 7337 ts_manager.cc:194] Registered new tserver with Master: dd36b520f02e4c7fa3082e278f14b414 (127.0.97.3:39649)
I20260929 20:47:23.625892 7808 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45167
I20260929 20:47:23.626021 7808 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:23.626452 7808 heartbeater.cc:511] Master 127.0.97.62:45167 requested a full tablet report, sending...
I20260929 20:47:23.626613 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:23.585077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46248 (local address 127.0.97.62:45167)
0929 20:47:23.606547 (+ 21470us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:23.606552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:23.606569 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:47:23.606630 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:23.606634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:23.606701 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:23.606805 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:23.608791 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.609831 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.612903 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.613623 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.613757 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.620264 (+ 6507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:23.620271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:23.620275 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:23.620308 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:23.623776 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.624474 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.624480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.624483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.624617 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.625027 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.625032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.625034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.625364 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:23.625557 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:23.626088 (+ 531us) server_negotiation.cc:299] Negotiation successful
0929 20:47:23.626283 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21357,"thread_start_us":21833,"threads_started":1}
I20260929 20:47:23.627367 6549 ts_manager.cc:194] Re-registered known tserver with Master: dd36b520f02e4c7fa3082e278f14b414 (127.0.97.3:39649)
I20260929 20:47:23.628609 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:23.585667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40578 (local address 127.0.97.61:42803)
0929 20:47:23.607307 (+ 21640us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:23.607312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:23.607327 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:47:23.607370 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:23.607375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:23.607434 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:23.609851 (+ 2417us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:23.610053 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.611142 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.614986 (+ 3844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.615625 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.616649 (+ 1024us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.619284 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:23.619292 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:23.619296 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:23.619330 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:23.625870 (+ 6540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.626551 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.626555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.626558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.626713 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.627171 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.627175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.627179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.627501 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:23.627710 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:23.628059 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 20:47:23.628257 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21531,"thread_start_us":25273,"threads_started":1}
I20260929 20:47:23.628625 6549 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51717
I20260929 20:47:23.631233 6549 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:47:23.638228 6549 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:47:23.663741 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:23.654780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41329 (local address 127.0.0.1:48118)
0929 20:47:23.655271 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:23.655290 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:23.655402 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:23.656422 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:23.656426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:23.656446 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:23.656770 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:23.656782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.658418 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.658424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:23.661093 (+ 2669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:23.661105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.662845 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.662851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:23.663074 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.663109 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 20:47:23.663215 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 20:47:23.663350 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":5,"thread_start_us":1045,"threads_started":1}
I20260929 20:47:23.663955 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:23.655356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.0.97.2:33431)
0929 20:47:23.655817 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:23.655826 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:23.656276 (+ 450us) server_negotiation.cc:407] Connection header received
0929 20:47:23.656539 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:23.656548 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:23.656797 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:23.656956 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:23.657641 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.659170 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.661512 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.662710 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.662790 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.662980 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:23.663566 (+ 586us) server_negotiation.cc:299] Negotiation successful
0929 20:47:23.663714 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":337,"threads_started":1}
I20260929 20:47:23.664384 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:23.655043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48118 (local address 127.0.97.1:41329)
0929 20:47:23.655817 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:23.655826 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:23.655854 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:47:23.655950 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:23.655958 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:23.656234 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:23.656973 (+ 739us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:23.657170 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.658286 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.661260 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.662649 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.662787 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.662980 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:23.663254 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 20:47:23.663423 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":524,"threads_started":1}
I20260929 20:47:23.665568 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:23.655593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33431 (local address 127.0.0.1:33674)
0929 20:47:23.656178 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:23.656194 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:23.656371 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:23.657169 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:23.657174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:23.657196 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:23.657482 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:23.657492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.659321 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.659326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:23.661363 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:23.661377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.663197 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.663205 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:23.663370 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.663402 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:47:23.664202 (+ 800us) client_negotiation.cc:240] Negotiation successful
0929 20:47:23.664320 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":27,"thread_start_us":1112,"threads_started":1}
I20260929 20:47:23.667218 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:23.656126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39649 (local address 127.0.0.1:40370)
0929 20:47:23.659731 (+ 3605us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:23.659748 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:23.659861 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:23.660747 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:23.660752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:23.660788 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:23.661169 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:23.661179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.662415 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.662421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:23.664503 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:23.664516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.666429 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.666436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:23.666592 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.666622 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:47:23.666758 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 20:47:23.666894 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3479,"thread_start_us":2895,"threads_started":1}
I20260929 20:47:23.667349 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:23.656482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.0.97.3:39649)
0929 20:47:23.658845 (+ 2363us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:23.658852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:23.659987 (+ 1135us) server_negotiation.cc:407] Connection header received
0929 20:47:23.660083 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:23.660090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:23.660331 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:23.660494 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:23.661341 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.662297 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.664761 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.666239 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.666361 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.666542 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:23.666843 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 20:47:23.666962 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2200,"thread_start_us":2019,"threads_started":1}
I20260929 20:47:23.668013 7450 tablet_service.cc:1467] Processing CreateTablet for tablet d1e83730e9264b298717bd7219a88ec7 (DEFAULT_TABLE table=test-table [id=63b3acd8a8ea419d85dcf5b884754d70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:23.669376 7450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e83730e9264b298717bd7219a88ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:23.669488 7601 tablet_service.cc:1467] Processing CreateTablet for tablet d1e83730e9264b298717bd7219a88ec7 (DEFAULT_TABLE table=test-table [id=63b3acd8a8ea419d85dcf5b884754d70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:23.670475 7601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e83730e9264b298717bd7219a88ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:23.670979 7742 tablet_service.cc:1467] Processing CreateTablet for tablet d1e83730e9264b298717bd7219a88ec7 (DEFAULT_TABLE table=test-table [id=63b3acd8a8ea419d85dcf5b884754d70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:23.672210 7742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e83730e9264b298717bd7219a88ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:23.685317 7842 tablet_bootstrap.cc:492] T d1e83730e9264b298717bd7219a88ec7 P d343b75e87f142f29eb74970160c325e: Bootstrap starting.
I20260929 20:47:23.686954 7841 tablet_bootstrap.cc:492] T d1e83730e9264b298717bd7219a88ec7 P 16a15205be8e418ba7b0856023a000ee: Bootstrap starting.
I20260929 20:47:23.688217 7843 tablet_bootstrap.cc:492] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414: Bootstrap starting.
I20260929 20:47:23.689432 7842 tablet_bootstrap.cc:654] T d1e83730e9264b298717bd7219a88ec7 P d343b75e87f142f29eb74970160c325e: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:23.690313 7841 tablet_bootstrap.cc:654] T d1e83730e9264b298717bd7219a88ec7 P 16a15205be8e418ba7b0856023a000ee: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:23.690668 7842 log.cc:824] T d1e83730e9264b298717bd7219a88ec7 P d343b75e87f142f29eb74970160c325e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:23.691409 7841 log.cc:824] T d1e83730e9264b298717bd7219a88ec7 P 16a15205be8e418ba7b0856023a000ee: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:23.691681 7843 tablet_bootstrap.cc:654] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:23.692772 7843 log.cc:824] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:23.694350 7842 tablet_bootstrap.cc:492] T d1e83730e9264b298717bd7219a88ec7 P d343b75e87f142f29eb74970160c325e: No bootstrap required, opened a new log
I20260929 20:47:23.694609 7842 ts_tablet_manager.cc:1397] T d1e83730e9264b298717bd7219a88ec7 P d343b75e87f142f29eb74970160c325e: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260929 20:47:23.695130 7841 tablet_bootstrap.cc:492] T d1e83730e9264b298717bd7219a88ec7 P 16a15205be8e418ba7b0856023a000ee: No bootstrap required, opened a new log
I20260929 20:47:23.695396 7841 ts_tablet_manager.cc:1397] T d1e83730e9264b298717bd7219a88ec7 P 16a15205be8e418ba7b0856023a000ee: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 20:47:23.696161 7843 tablet_bootstrap.cc:492] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414: No bootstrap required, opened a new log
I20260929 20:47:23.696400 7843 ts_tablet_manager.cc:1397] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 20:47:23.703517 7841 raft_consensus.cc:348] T d1e83730e9264b298717bd7219a88ec7 P 16a15205be8e418ba7b0856023a000ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343b75e87f142f29eb74970160c325e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41329 } } peers { permanent_uuid: "16a15205be8e418ba7b0856023a000ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33431 } } peers { permanent_uuid: "dd36b520f02e4c7fa3082e278f14b414" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39649 } }
I20260929 20:47:23.703943 7841 raft_consensus.cc:374] T d1e83730e9264b298717bd7219a88ec7 P 16a15205be8e418ba7b0856023a000ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:23.704038 7841 raft_consensus.cc:729] T d1e83730e9264b298717bd7219a88ec7 P 16a15205be8e418ba7b0856023a000ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a15205be8e418ba7b0856023a000ee, State: Initialized, Role: FOLLOWER
I20260929 20:47:23.704613 7841 consensus_queue.cc:260] T d1e83730e9264b298717bd7219a88ec7 P 16a15205be8e418ba7b0856023a000ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343b75e87f142f29eb74970160c325e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41329 } } peers { permanent_uuid: "16a15205be8e418ba7b0856023a000ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33431 } } peers { permanent_uuid: "dd36b520f02e4c7fa3082e278f14b414" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39649 } }
I20260929 20:47:23.705837 7842 raft_consensus.cc:348] T d1e83730e9264b298717bd7219a88ec7 P d343b75e87f142f29eb74970160c325e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343b75e87f142f29eb74970160c325e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41329 } } peers { permanent_uuid: "16a15205be8e418ba7b0856023a000ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33431 } } peers { permanent_uuid: "dd36b520f02e4c7fa3082e278f14b414" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39649 } }
I20260929 20:47:23.706260 7842 raft_consensus.cc:374] T d1e83730e9264b298717bd7219a88ec7 P d343b75e87f142f29eb74970160c325e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:23.706370 7842 raft_consensus.cc:729] T d1e83730e9264b298717bd7219a88ec7 P d343b75e87f142f29eb74970160c325e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d343b75e87f142f29eb74970160c325e, State: Initialized, Role: FOLLOWER
I20260929 20:47:23.706336 7843 raft_consensus.cc:348] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343b75e87f142f29eb74970160c325e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41329 } } peers { permanent_uuid: "16a15205be8e418ba7b0856023a000ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33431 } } peers { permanent_uuid: "dd36b520f02e4c7fa3082e278f14b414" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39649 } }
I20260929 20:47:23.706733 7843 raft_consensus.cc:374] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:23.706828 7843 raft_consensus.cc:729] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd36b520f02e4c7fa3082e278f14b414, State: Initialized, Role: FOLLOWER
I20260929 20:47:23.706936 7842 consensus_queue.cc:260] T d1e83730e9264b298717bd7219a88ec7 P d343b75e87f142f29eb74970160c325e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343b75e87f142f29eb74970160c325e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41329 } } peers { permanent_uuid: "16a15205be8e418ba7b0856023a000ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33431 } } peers { permanent_uuid: "dd36b520f02e4c7fa3082e278f14b414" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39649 } }
I20260929 20:47:23.707365 7843 consensus_queue.cc:260] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343b75e87f142f29eb74970160c325e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41329 } } peers { permanent_uuid: "16a15205be8e418ba7b0856023a000ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33431 } } peers { permanent_uuid: "dd36b520f02e4c7fa3082e278f14b414" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39649 } }
I20260929 20:47:23.708570 7841 ts_tablet_manager.cc:1428] T d1e83730e9264b298717bd7219a88ec7 P 16a15205be8e418ba7b0856023a000ee: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260929 20:47:23.708896 7667 heartbeater.cc:503] Master 127.0.97.62:45167 was elected leader, sending a full tablet report...
I20260929 20:47:23.716136 7842 ts_tablet_manager.cc:1428] T d1e83730e9264b298717bd7219a88ec7 P d343b75e87f142f29eb74970160c325e: Time spent starting tablet: real 0.021s user 0.014s sys 0.005s
I20260929 20:47:23.718158 7843 ts_tablet_manager.cc:1428] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414: Time spent starting tablet: real 0.022s user 0.020s sys 0.000s
I20260929 20:47:23.719071 7808 heartbeater.cc:503] Master 127.0.97.62:45167 was elected leader, sending a full tablet report...
W20260929 20:47:23.750025 7675 tablet.cc:2369] T d1e83730e9264b298717bd7219a88ec7 P 16a15205be8e418ba7b0856023a000ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:47:23.793016 7849 raft_consensus.cc:482] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:47:23.793308 7849 raft_consensus.cc:504] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343b75e87f142f29eb74970160c325e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41329 } } peers { permanent_uuid: "16a15205be8e418ba7b0856023a000ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33431 } } peers { permanent_uuid: "dd36b520f02e4c7fa3082e278f14b414" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39649 } }
I20260929 20:47:23.794925 7849 leader_election.cc:288] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d343b75e87f142f29eb74970160c325e (127.0.97.1:41329), 16a15205be8e418ba7b0856023a000ee (127.0.97.2:33431)
I20260929 20:47:23.802076 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:23.795278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39523 (local address 127.0.97.1:41329)
0929 20:47:23.795457 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:23.795462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:23.795483 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:47:23.795674 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:23.795679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:23.795753 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:23.795852 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:23.796325 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.797389 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.799914 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.801112 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.801237 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.801332 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:23.801706 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 20:47:23.801831 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:47:23.802112 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:23.795305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39373 (local address 127.0.97.2:33431)
0929 20:47:23.795458 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:23.795463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:23.795483 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:47:23.795674 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:23.795679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:23.795757 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:23.795852 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:23.796325 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.797389 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.799908 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.801129 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.801237 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.801329 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:23.801707 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 20:47:23.801830 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 20:47:23.802241 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:23.795151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33431 (local address 127.0.97.3:39373)
0929 20:47:23.795404 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:23.795424 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:23.795543 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:23.795869 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:23.795874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:23.795908 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:23.796194 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:23.796202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.797519 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.797523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:23.799743 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:23.799758 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.801266 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.801272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:23.801502 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.801587 (+ 85us) client_negotiation.cc:769] Sending connection context
0929 20:47:23.801680 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 20:47:23.801845 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":217,"spinlock_wait_cycles":16512}
I20260929 20:47:23.802273 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:23.795240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41329 (local address 127.0.97.3:39523)
0929 20:47:23.795415 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:23.795428 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:23.795543 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:23.795869 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:23.795874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:23.795905 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:23.796190 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:23.796201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.797512 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.797517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:23.799746 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:23.799758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.801267 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.801272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:23.801539 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.801592 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 20:47:23.801680 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 20:47:23.801845 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":182,"spinlock_wait_cycles":10496}
I20260929 20:47:23.803295 7470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e83730e9264b298717bd7219a88ec7" candidate_uuid: "dd36b520f02e4c7fa3082e278f14b414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d343b75e87f142f29eb74970160c325e" is_pre_election: true
I20260929 20:47:23.803296 7621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e83730e9264b298717bd7219a88ec7" candidate_uuid: "dd36b520f02e4c7fa3082e278f14b414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16a15205be8e418ba7b0856023a000ee" is_pre_election: true
I20260929 20:47:23.803761 7621 raft_consensus.cc:2454] T d1e83730e9264b298717bd7219a88ec7 P 16a15205be8e418ba7b0856023a000ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd36b520f02e4c7fa3082e278f14b414 in term 0.
I20260929 20:47:23.803772 7470 raft_consensus.cc:2454] T d1e83730e9264b298717bd7219a88ec7 P d343b75e87f142f29eb74970160c325e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd36b520f02e4c7fa3082e278f14b414 in term 0.
I20260929 20:47:23.804344 7697 leader_election.cc:302] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16a15205be8e418ba7b0856023a000ee, dd36b520f02e4c7fa3082e278f14b414; no voters:
I20260929 20:47:23.804728 7849 raft_consensus.cc:2781] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:47:23.804847 7849 raft_consensus.cc:482] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:47:23.804934 7849 raft_consensus.cc:3037] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:23.807513 7849 raft_consensus.cc:504] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343b75e87f142f29eb74970160c325e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41329 } } peers { permanent_uuid: "16a15205be8e418ba7b0856023a000ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33431 } } peers { permanent_uuid: "dd36b520f02e4c7fa3082e278f14b414" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39649 } }
I20260929 20:47:23.808044 7849 leader_election.cc:288] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414 [CANDIDATE]: Term 1 election: Requested vote from peers d343b75e87f142f29eb74970160c325e (127.0.97.1:41329), 16a15205be8e418ba7b0856023a000ee (127.0.97.2:33431)
I20260929 20:47:23.808488 7470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e83730e9264b298717bd7219a88ec7" candidate_uuid: "dd36b520f02e4c7fa3082e278f14b414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d343b75e87f142f29eb74970160c325e"
I20260929 20:47:23.808501 7621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e83730e9264b298717bd7219a88ec7" candidate_uuid: "dd36b520f02e4c7fa3082e278f14b414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16a15205be8e418ba7b0856023a000ee"
I20260929 20:47:23.808653 7470 raft_consensus.cc:3037] T d1e83730e9264b298717bd7219a88ec7 P d343b75e87f142f29eb74970160c325e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:23.808699 7621 raft_consensus.cc:3037] T d1e83730e9264b298717bd7219a88ec7 P 16a15205be8e418ba7b0856023a000ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:23.810904 7470 raft_consensus.cc:2454] T d1e83730e9264b298717bd7219a88ec7 P d343b75e87f142f29eb74970160c325e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd36b520f02e4c7fa3082e278f14b414 in term 1.
I20260929 20:47:23.811067 7621 raft_consensus.cc:2454] T d1e83730e9264b298717bd7219a88ec7 P 16a15205be8e418ba7b0856023a000ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd36b520f02e4c7fa3082e278f14b414 in term 1.
I20260929 20:47:23.811338 7698 leader_election.cc:302] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d343b75e87f142f29eb74970160c325e, dd36b520f02e4c7fa3082e278f14b414; no voters:
I20260929 20:47:23.811618 7849 raft_consensus.cc:2781] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:47:23.812000 7849 raft_consensus.cc:686] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414 [term 1 LEADER]: Becoming Leader. State: Replica: dd36b520f02e4c7fa3082e278f14b414, State: Running, Role: LEADER
I20260929 20:47:23.812479 7849 consensus_queue.cc:237] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343b75e87f142f29eb74970160c325e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41329 } } peers { permanent_uuid: "16a15205be8e418ba7b0856023a000ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33431 } } peers { permanent_uuid: "dd36b520f02e4c7fa3082e278f14b414" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39649 } }
W20260929 20:47:23.816301 7810 tablet.cc:2369] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:47:23.817346 6548 catalog_manager.cc:5495] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd36b520f02e4c7fa3082e278f14b414 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "dd36b520f02e4c7fa3082e278f14b414" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d343b75e87f142f29eb74970160c325e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16a15205be8e418ba7b0856023a000ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd36b520f02e4c7fa3082e278f14b414" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39649 } health_report { overall_health: HEALTHY } } }
I20260929 20:47:23.853063 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:23.848239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40384 (local address 127.0.97.3:39649)
0929 20:47:23.848429 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:23.848435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:23.848456 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:47:23.848588 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:23.848593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:23.848699 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:23.848850 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:47:23.849203 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.849867 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.851149 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.851541 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.851644 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.852078 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:47:23.852555 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:47:23.852679 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:23.852766 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 20:47:23.852835 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 20:47:23.860476 7621 raft_consensus.cc:1261] T d1e83730e9264b298717bd7219a88ec7 P 16a15205be8e418ba7b0856023a000ee [term 1 FOLLOWER]: Refusing update from remote peer dd36b520f02e4c7fa3082e278f14b414: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:47:23.860476 7470 raft_consensus.cc:1261] T d1e83730e9264b298717bd7219a88ec7 P d343b75e87f142f29eb74970160c325e [term 1 FOLLOWER]: Refusing update from remote peer dd36b520f02e4c7fa3082e278f14b414: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:47:23.861297 7850 consensus_queue.cc:1035] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a15205be8e418ba7b0856023a000ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:47:23.861632 7849 consensus_queue.cc:1035] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d343b75e87f142f29eb74970160c325e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:47:23.872850 7853 mvcc.cc:205] Tried to move back new op lower bound from 7334768000443400192 to 7334768000259469312. Current Snapshot: MvccSnapshot[applied={T|T < 7334768000443400192}]
I20260929 20:47:23.888243 6549 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:47:23.888540 6549 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:47:23.894676 6549 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1460986649014da8aff571f52348fcad: Sending DeleteTablet for 3 replicas of tablet d1e83730e9264b298717bd7219a88ec7
I20260929 20:47:23.895809 7601 tablet_service.cc:1514] Processing DeleteTablet for tablet d1e83730e9264b298717bd7219a88ec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:23 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33674
I20260929 20:47:23.896155 7742 tablet_service.cc:1514] Processing DeleteTablet for tablet d1e83730e9264b298717bd7219a88ec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:23 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40370
I20260929 20:47:23.896474 7450 tablet_service.cc:1514] Processing DeleteTablet for tablet d1e83730e9264b298717bd7219a88ec7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:23 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48118
I20260929 20:47:23.896682 7862 tablet_replica.cc:330] T d1e83730e9264b298717bd7219a88ec7 P 16a15205be8e418ba7b0856023a000ee: stopping tablet replica
I20260929 20:47:23.897527 7863 tablet_replica.cc:330] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414: stopping tablet replica
I20260929 20:47:23.898337 7863 raft_consensus.cc:2229] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:47:23.899183 7863 raft_consensus.cc:2258] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:23.900851 7862 raft_consensus.cc:2229] T d1e83730e9264b298717bd7219a88ec7 P 16a15205be8e418ba7b0856023a000ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:47:23.900942 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 6517
I20260929 20:47:23.901746 7862 raft_consensus.cc:2258] T d1e83730e9264b298717bd7219a88ec7 P 16a15205be8e418ba7b0856023a000ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:23.901835 6579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:23.902840 7864 tablet_replica.cc:330] T d1e83730e9264b298717bd7219a88ec7 P d343b75e87f142f29eb74970160c325e: stopping tablet replica
I20260929 20:47:23.903223 7864 raft_consensus.cc:2229] T d1e83730e9264b298717bd7219a88ec7 P d343b75e87f142f29eb74970160c325e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:47:23.903638 7864 raft_consensus.cc:2258] T d1e83730e9264b298717bd7219a88ec7 P d343b75e87f142f29eb74970160c325e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:23.904072 7862 ts_tablet_manager.cc:1905] T d1e83730e9264b298717bd7219a88ec7 P 16a15205be8e418ba7b0856023a000ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:23.907114 7863 ts_tablet_manager.cc:1905] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:23.907235 7864 ts_tablet_manager.cc:1905] T d1e83730e9264b298717bd7219a88ec7 P d343b75e87f142f29eb74970160c325e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:23.911520 7862 ts_tablet_manager.cc:1918] T d1e83730e9264b298717bd7219a88ec7 P 16a15205be8e418ba7b0856023a000ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:47:23.911736 7862 log.cc:1196] T d1e83730e9264b298717bd7219a88ec7 P 16a15205be8e418ba7b0856023a000ee: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/d1e83730e9264b298717bd7219a88ec7
I20260929 20:47:23.912263 7862 ts_tablet_manager.cc:1939] T d1e83730e9264b298717bd7219a88ec7 P 16a15205be8e418ba7b0856023a000ee: Deleting consensus metadata
I20260929 20:47:23.914142 7864 ts_tablet_manager.cc:1918] T d1e83730e9264b298717bd7219a88ec7 P d343b75e87f142f29eb74970160c325e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:47:23.914324 7864 log.cc:1196] T d1e83730e9264b298717bd7219a88ec7 P d343b75e87f142f29eb74970160c325e: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/d1e83730e9264b298717bd7219a88ec7
I20260929 20:47:23.915067 7864 ts_tablet_manager.cc:1939] T d1e83730e9264b298717bd7219a88ec7 P d343b75e87f142f29eb74970160c325e: Deleting consensus metadata
I20260929 20:47:23.920267 7863 ts_tablet_manager.cc:1918] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:47:23.920529 7863 log.cc:1196] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/d1e83730e9264b298717bd7219a88ec7
I20260929 20:47:23.921094 7863 ts_tablet_manager.cc:1939] T d1e83730e9264b298717bd7219a88ec7 P dd36b520f02e4c7fa3082e278f14b414: Deleting consensus metadata
I20260929 20:47:23.929612 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:23.897755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40590 (local address 127.0.97.61:42803)
0929 20:47:23.897931 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:23.897938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:23.900023 (+ 2085us) server_negotiation.cc:407] Connection header received
0929 20:47:23.907847 (+ 7824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:23.907852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:23.907930 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:23.908029 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:23.914347 (+ 6318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.915495 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.918657 (+ 3162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.919277 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.919395 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.922872 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:23.922878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:23.922883 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:23.922920 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:23.926781 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.927616 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.927622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.927625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.927773 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.928158 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.928163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.928166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.928469 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:23.928670 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:23.929055 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 20:47:23.929280 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:47:23.944922 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:23.900497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38920 (local address 127.0.97.60:46467)
0929 20:47:23.900667 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:23.900673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:23.929728 (+ 29055us) server_negotiation.cc:407] Connection header received
0929 20:47:23.929976 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:23.929982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:23.930063 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:23.930130 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:23.931936 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.933000 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.934775 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:23.935437 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:23.935549 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:23.938843 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:23.938849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:23.938854 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:23.938890 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:23.942198 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.942875 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.942881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.942884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.943078 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:23.943477 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:23.943481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:23.943485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:23.943805 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:23.944031 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:23.944353 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 20:47:23.944559 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 20:47:24.045814 6536 catalog_manager.cc:4841] TS dd36b520f02e4c7fa3082e278f14b414 (127.0.97.3:39649): tablet d1e83730e9264b298717bd7219a88ec7 (table test-table [id=63b3acd8a8ea419d85dcf5b884754d70]) successfully deleted
I20260929 20:47:24.046482 6536 catalog_manager.cc:4841] TS d343b75e87f142f29eb74970160c325e (127.0.97.1:41329): tablet d1e83730e9264b298717bd7219a88ec7 (table test-table [id=63b3acd8a8ea419d85dcf5b884754d70]) successfully deleted
I20260929 20:47:24.046459 6535 catalog_manager.cc:4841] TS 16a15205be8e418ba7b0856023a000ee (127.0.97.2:33431): tablet d1e83730e9264b298717bd7219a88ec7 (table test-table [id=63b3acd8a8ea419d85dcf5b884754d70]) successfully deleted
W20260929 20:47:24.046763 6535 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.2:33431, user_credentials={real_user=slave}} blocked reactor thread for 132686us
I20260929 20:47:24.047187 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 6517
I20260929 20:47:24.085356 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:24.068396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38934 (local address 127.0.97.60:46467)
0929 20:47:24.068585 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:24.068592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:24.069814 (+ 1222us) server_negotiation.cc:407] Connection header received
0929 20:47:24.070081 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:24.070087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:24.070186 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:24.070263 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:24.072871 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:24.074047 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:24.076185 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:24.076869 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:24.076969 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:24.080207 (+ 3238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:24.080213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:24.080217 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:24.080249 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:24.082969 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:24.083648 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:24.083653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:24.083657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:24.083765 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:24.084110 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:24.084116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:24.084119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:24.084377 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:24.084596 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:24.084898 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 20:47:24.085096 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:47:24.090188 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:24.069497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40602 (local address 127.0.97.61:42803)
0929 20:47:24.069692 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:24.069698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:24.075436 (+ 5738us) server_negotiation.cc:407] Connection header received
0929 20:47:24.075883 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:24.075890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:24.075976 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:24.076041 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:24.078544 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:24.079674 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:24.081463 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:24.082038 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:24.082160 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:24.084956 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:24.084961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:24.084965 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:24.085002 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:24.087868 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:24.088550 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:24.088555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:24.088558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:24.088666 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:24.089000 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:24.089002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:24.089004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:24.089269 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:24.089415 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:24.089744 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 20:47:24.089878 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 20:47:24.877885 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:24.876990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.97.3:48769)
0929 20:47:24.877509 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:24.877669 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":272,"threads_started":1}
W20260929 20:47:24.878319 7808 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45167 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
I20260929 20:47:24.880614 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:24.880007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.97.2:43295)
0929 20:47:24.880375 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:24.880468 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":196,"threads_started":1}
W20260929 20:47:24.880930 7667 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45167 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
I20260929 20:47:24.882390 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:24.881480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.97.1:36487)
0929 20:47:24.882135 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:24.882246 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":470,"threads_started":1}
W20260929 20:47:24.883002 7516 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45167 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
I20260929 20:47:25.395466 7877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1460986649014da8aff571f52348fcad)
I20260929 20:47:25.395749 7878 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1460986649014da8aff571f52348fcad)
I20260929 20:47:25.395737 7877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: false } }
I20260929 20:47:25.395969 7878 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: false } }
I20260929 20:47:25.397068 7877 leader_election.cc:288] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1460986649014da8aff571f52348fcad (127.0.97.62:45167), ee15b91d5caa47b6b8021324f9fc53b1 (127.0.97.60:46467)
I20260929 20:47:25.397150 7878 leader_election.cc:288] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1460986649014da8aff571f52348fcad (127.0.97.62:45167), 1097f58cae6a4662999174443b82b300 (127.0.97.61:42803)
I20260929 20:47:25.404412 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.397286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.0.1:46736)
0929 20:47:25.404006 (+ 6720us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.404194 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6590,"threads_started":1}
I20260929 20:47:25.404903 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.397546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.0.1:46740)
0929 20:47:25.404406 (+ 6860us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.404683 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6780,"thread_start_us":6680,"threads_started":1}
W20260929 20:47:25.409555 7323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
W20260929 20:47:25.412904 7134 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
I20260929 20:47:25.413528 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.397472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46467 (local address 127.0.0.1:38938)
0929 20:47:25.404890 (+ 7418us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.404909 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:25.405054 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:25.405636 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:25.405642 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:25.405668 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:25.405959 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:25.405970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.407587 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.407594 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:25.409803 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:25.409820 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.412754 (+ 2934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.412763 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:25.412919 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:25.412951 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:47:25.413048 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 20:47:25.413195 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7330,"thread_start_us":7087,"threads_started":1}
I20260929 20:47:25.413801 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.397449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38938 (local address 127.0.97.60:46467)
0929 20:47:25.404087 (+ 6638us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:25.404093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:25.405190 (+ 1097us) server_negotiation.cc:407] Connection header received
0929 20:47:25.405267 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:25.405272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:25.405355 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:25.405438 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:25.406149 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.407392 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.409993 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.412417 (+ 2424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.412529 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:25.412641 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:25.413370 (+ 729us) server_negotiation.cc:299] Negotiation successful
0929 20:47:25.413515 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6511,"thread_start_us":7061,"threads_started":1}
I20260929 20:47:25.414440 7347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1097f58cae6a4662999174443b82b300" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" is_pre_election: true
I20260929 20:47:25.414750 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.413438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.0.1:46758)
0929 20:47:25.414301 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.414475 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":581,"threads_started":1}
I20260929 20:47:25.414804 7347 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1097f58cae6a4662999174443b82b300 in term 1.
W20260929 20:47:25.414898 7323 leader_election.cc:334] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1460986649014da8aff571f52348fcad (127.0.97.62:45167): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
I20260929 20:47:25.415531 7135 leader_election.cc:302] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1097f58cae6a4662999174443b82b300, ee15b91d5caa47b6b8021324f9fc53b1; no voters:
I20260929 20:47:25.415925 7877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 20:47:25.416124 7877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1460986649014da8aff571f52348fcad)
I20260929 20:47:25.416239 7877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:47:25.416397 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.416034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.0.1:46766)
0929 20:47:25.416155 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.416237 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 20:47:25.416646 7134 leader_election.cc:334] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1460986649014da8aff571f52348fcad (127.0.97.62:45167): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
I20260929 20:47:25.418545 7877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: false } }
I20260929 20:47:25.419173 7877 leader_election.cc:288] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [CANDIDATE]: Term 2 election: Requested vote from peers 1460986649014da8aff571f52348fcad (127.0.97.62:45167), ee15b91d5caa47b6b8021324f9fc53b1 (127.0.97.60:46467)
I20260929 20:47:25.419585 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.419219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.0.1:46768)
0929 20:47:25.419341 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.419428 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 20:47:25.419853 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.409284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42803 (local address 127.0.0.1:40618)
0929 20:47:25.409433 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.409451 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:25.409590 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:25.414988 (+ 5398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:25.414993 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:25.415042 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:25.415279 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:25.415290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.416823 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.416828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:25.418021 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:25.418032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.419172 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.419177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:25.419345 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:25.419377 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:47:25.419453 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 20:47:25.419581 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 20:47:25.419814 7347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1097f58cae6a4662999174443b82b300" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ee15b91d5caa47b6b8021324f9fc53b1"
I20260929 20:47:25.419906 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.404652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.0.97.61:42803)
0929 20:47:25.414677 (+ 10025us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:25.414683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:25.414700 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:47:25.414759 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:25.414763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:25.414832 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:25.415399 (+ 567us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:25.415639 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.416700 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.418167 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.418992 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.419139 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:25.419288 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:25.419492 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 20:47:25.419613 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9916,"thread_start_us":9718,"threads_started":1}
I20260929 20:47:25.420007 7347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:47:25.420691 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.420311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.0.1:46782)
0929 20:47:25.420471 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.420543 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 20:47:25.420897 7158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1097f58cae6a4662999174443b82b300" is_pre_election: true
W20260929 20:47:25.420925 7134 leader_election.cc:334] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1460986649014da8aff571f52348fcad (127.0.97.62:45167): Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
I20260929 20:47:25.421186 7158 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee15b91d5caa47b6b8021324f9fc53b1 in current term 2: Already voted for candidate 1097f58cae6a4662999174443b82b300 in this term.
I20260929 20:47:25.421646 7323 leader_election.cc:302] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee15b91d5caa47b6b8021324f9fc53b1; no voters: 1097f58cae6a4662999174443b82b300, 1460986649014da8aff571f52348fcad
I20260929 20:47:25.422442 7347 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1097f58cae6a4662999174443b82b300 in term 2.
I20260929 20:47:25.422600 7878 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:47:25.422834 7135 leader_election.cc:302] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1097f58cae6a4662999174443b82b300, ee15b91d5caa47b6b8021324f9fc53b1; no voters: 1460986649014da8aff571f52348fcad
I20260929 20:47:25.423146 7877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:47:25.423516 7877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [term 2 LEADER]: Becoming Leader. State: Replica: 1097f58cae6a4662999174443b82b300, State: Running, Role: LEADER
I20260929 20:47:25.423852 7877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: false } }
I20260929 20:47:25.425331 7887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1097f58cae6a4662999174443b82b300. Latest consensus state: current_term: 2 leader_uuid: "1097f58cae6a4662999174443b82b300" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: false } } }
I20260929 20:47:25.425709 7887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:25.426337 7889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:47:25.429759 7889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:47:25.430497 7889 catalog_manager.cc:1259] Loaded cluster ID: 48312dabe6844bb0a0147505a1576ca1
I20260929 20:47:25.430567 7889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:47:25.431671 7889 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:47:25.432420 7889 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300: Loaded TSK: 0
I20260929 20:47:25.433187 7889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:47:25.465649 7148 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:47:25.472330 7148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:47:25.476257 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.475892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.0.1:46806)
0929 20:47:25.476043 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.476135 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 20:47:25.476418 7347 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [term 2 FOLLOWER]: Refusing update from remote peer 1097f58cae6a4662999174443b82b300: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 20:47:25.477190 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.476812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.0.1:46812)
0929 20:47:25.476965 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.477061 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 20:47:25.477129 7887 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 20:47:25.477375 7134 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 -> Peer 1460986649014da8aff571f52348fcad (127.0.97.62:45167): Couldn't send request to peer 1460986649014da8aff571f52348fcad. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 20:47:25.480113 7878 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1097f58cae6a4662999174443b82b300. Latest consensus state: current_term: 2 leader_uuid: "1097f58cae6a4662999174443b82b300" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: false } } }
I20260929 20:47:25.480406 7878 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:47:25.480727 7890 mvcc.cc:205] Tried to move back new op lower bound from 7334768007064903680 to 7334768006859141120. Current Snapshot: MvccSnapshot[applied={T|T < 7334768007064903680}]
I20260929 20:47:25.481178 7887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "1097f58cae6a4662999174443b82b300" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: false } } }
I20260929 20:47:25.481441 7887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300 [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:25.481681 7878 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "1097f58cae6a4662999174443b82b300" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "1460986649014da8aff571f52348fcad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45167 } } peers { permanent_uuid: "1097f58cae6a4662999174443b82b300" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42803 } attrs { promote: false } } peers { permanent_uuid: "ee15b91d5caa47b6b8021324f9fc53b1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46467 } attrs { promote: false } } }
I20260929 20:47:25.481863 7891 mvcc.cc:205] Tried to move back new op lower bound from 7334768007064903680 to 7334768006859141120. Current Snapshot: MvccSnapshot[applied={T|T < 7334768007064903680}]
I20260929 20:47:25.481909 7878 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee15b91d5caa47b6b8021324f9fc53b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:47:25.495411 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.488608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33431 (local address 127.0.0.1:33680)
0929 20:47:25.488808 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.488826 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:25.488924 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:25.489454 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:25.489459 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:25.489484 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:25.489682 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:25.489694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.491234 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.491238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:25.493157 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:25.493173 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.494683 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.494689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:25.494848 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:25.494883 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 20:47:25.494976 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 20:47:25.495121 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 20:47:25.495476 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.488735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.0.97.2:33431)
0929 20:47:25.489124 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:25.489129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:25.489147 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:47:25.489211 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:25.489215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:25.489284 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:25.489424 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:25.489838 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.491052 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.493362 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.494528 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.494647 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:25.494739 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:25.495008 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 20:47:25.495168 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":209,"threads_started":1}
I20260929 20:47:25.496397 7601 tablet_service.cc:1467] Processing CreateTablet for tablet 0baa383f68dc4d4f9659115e503ae420 (DEFAULT_TABLE table=test-table [id=ed13ad33ae9c4db3bb55ec31531dc241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:25.496578 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.489635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39649 (local address 127.0.0.1:40390)
0929 20:47:25.490164 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.490180 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:25.490272 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:25.490993 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:25.490998 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:25.491042 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:25.491447 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:25.491458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.492597 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.492602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:25.494381 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:25.494391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.495838 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.495843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:25.495995 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:25.496026 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:47:25.496086 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 20:47:25.496205 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":1,"thread_start_us":746,"threads_started":1}
I20260929 20:47:25.496852 7601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0baa383f68dc4d4f9659115e503ae420. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:25.497274 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.490305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.0.97.3:39649)
0929 20:47:25.490695 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:25.490701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:25.490718 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:47:25.490780 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:25.490785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:25.490852 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:25.491597 (+ 745us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:25.491723 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.492489 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.494514 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.495623 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.496757 (+ 1134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:25.496854 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:25.496905 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 20:47:25.497021 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":491,"threads_started":1}
I20260929 20:47:25.497304 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.489225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41329 (local address 127.0.0.1:48128)
0929 20:47:25.489592 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.489608 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:25.489691 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:25.491508 (+ 1817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:25.491513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:25.491537 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:25.491899 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:25.491909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.493312 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.493316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:25.495294 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:25.495308 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.496628 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.496635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:25.496812 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:25.496844 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:47:25.496902 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 20:47:25.497023 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245}
I20260929 20:47:25.497921 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.489341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48128 (local address 127.0.97.1:41329)
0929 20:47:25.491209 (+ 1868us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:25.491216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:25.491233 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:47:25.491302 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:25.491307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:25.491374 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:25.491473 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:25.492124 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.493161 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.495491 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.496384 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.497485 (+ 1101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:25.497576 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:25.497615 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 20:47:25.497705 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1740,"thread_start_us":1630,"threads_started":1}
I20260929 20:47:25.498471 7450 tablet_service.cc:1467] Processing CreateTablet for tablet 0baa383f68dc4d4f9659115e503ae420 (DEFAULT_TABLE table=test-table [id=ed13ad33ae9c4db3bb55ec31531dc241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:25.498792 7450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0baa383f68dc4d4f9659115e503ae420. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:25.499651 7742 tablet_service.cc:1467] Processing CreateTablet for tablet 0baa383f68dc4d4f9659115e503ae420 (DEFAULT_TABLE table=test-table [id=ed13ad33ae9c4db3bb55ec31531dc241]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:25.500023 7742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0baa383f68dc4d4f9659115e503ae420. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:25.502110 7900 tablet_bootstrap.cc:492] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee: Bootstrap starting.
I20260929 20:47:25.503929 7900 tablet_bootstrap.cc:654] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:25.504611 7901 tablet_bootstrap.cc:492] T 0baa383f68dc4d4f9659115e503ae420 P d343b75e87f142f29eb74970160c325e: Bootstrap starting.
I20260929 20:47:25.506352 7901 tablet_bootstrap.cc:654] T 0baa383f68dc4d4f9659115e503ae420 P d343b75e87f142f29eb74970160c325e: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:25.506531 7900 tablet_bootstrap.cc:492] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee: No bootstrap required, opened a new log
I20260929 20:47:25.506659 7900 ts_tablet_manager.cc:1397] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260929 20:47:25.507444 7903 tablet_bootstrap.cc:492] T 0baa383f68dc4d4f9659115e503ae420 P dd36b520f02e4c7fa3082e278f14b414: Bootstrap starting.
I20260929 20:47:25.507342 7900 raft_consensus.cc:348] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a15205be8e418ba7b0856023a000ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33431 } } peers { permanent_uuid: "d343b75e87f142f29eb74970160c325e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41329 } } peers { permanent_uuid: "dd36b520f02e4c7fa3082e278f14b414" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39649 } }
I20260929 20:47:25.507572 7900 raft_consensus.cc:374] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:25.507692 7900 raft_consensus.cc:729] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a15205be8e418ba7b0856023a000ee, State: Initialized, Role: FOLLOWER
I20260929 20:47:25.508036 7900 consensus_queue.cc:260] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a15205be8e418ba7b0856023a000ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33431 } } peers { permanent_uuid: "d343b75e87f142f29eb74970160c325e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41329 } } peers { permanent_uuid: "dd36b520f02e4c7fa3082e278f14b414" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39649 } }
I20260929 20:47:25.508744 7901 tablet_bootstrap.cc:492] T 0baa383f68dc4d4f9659115e503ae420 P d343b75e87f142f29eb74970160c325e: No bootstrap required, opened a new log
I20260929 20:47:25.508863 7901 ts_tablet_manager.cc:1397] T 0baa383f68dc4d4f9659115e503ae420 P d343b75e87f142f29eb74970160c325e: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260929 20:47:25.509306 7903 tablet_bootstrap.cc:654] T 0baa383f68dc4d4f9659115e503ae420 P dd36b520f02e4c7fa3082e278f14b414: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:25.509616 7901 raft_consensus.cc:348] T 0baa383f68dc4d4f9659115e503ae420 P d343b75e87f142f29eb74970160c325e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a15205be8e418ba7b0856023a000ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33431 } } peers { permanent_uuid: "d343b75e87f142f29eb74970160c325e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41329 } } peers { permanent_uuid: "dd36b520f02e4c7fa3082e278f14b414" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39649 } }
I20260929 20:47:25.509788 7901 raft_consensus.cc:374] T 0baa383f68dc4d4f9659115e503ae420 P d343b75e87f142f29eb74970160c325e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:25.509896 7901 raft_consensus.cc:729] T 0baa383f68dc4d4f9659115e503ae420 P d343b75e87f142f29eb74970160c325e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d343b75e87f142f29eb74970160c325e, State: Initialized, Role: FOLLOWER
I20260929 20:47:25.510121 7901 consensus_queue.cc:260] T 0baa383f68dc4d4f9659115e503ae420 P d343b75e87f142f29eb74970160c325e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a15205be8e418ba7b0856023a000ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33431 } } peers { permanent_uuid: "d343b75e87f142f29eb74970160c325e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41329 } } peers { permanent_uuid: "dd36b520f02e4c7fa3082e278f14b414" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39649 } }
I20260929 20:47:25.510941 7901 ts_tablet_manager.cc:1428] T 0baa383f68dc4d4f9659115e503ae420 P d343b75e87f142f29eb74970160c325e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:47:25.511346 7903 tablet_bootstrap.cc:492] T 0baa383f68dc4d4f9659115e503ae420 P dd36b520f02e4c7fa3082e278f14b414: No bootstrap required, opened a new log
I20260929 20:47:25.511508 7903 ts_tablet_manager.cc:1397] T 0baa383f68dc4d4f9659115e503ae420 P dd36b520f02e4c7fa3082e278f14b414: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260929 20:47:25.512287 7903 raft_consensus.cc:348] T 0baa383f68dc4d4f9659115e503ae420 P dd36b520f02e4c7fa3082e278f14b414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a15205be8e418ba7b0856023a000ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33431 } } peers { permanent_uuid: "d343b75e87f142f29eb74970160c325e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41329 } } peers { permanent_uuid: "dd36b520f02e4c7fa3082e278f14b414" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39649 } }
I20260929 20:47:25.512573 7903 raft_consensus.cc:374] T 0baa383f68dc4d4f9659115e503ae420 P dd36b520f02e4c7fa3082e278f14b414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:25.512696 7903 raft_consensus.cc:729] T 0baa383f68dc4d4f9659115e503ae420 P dd36b520f02e4c7fa3082e278f14b414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd36b520f02e4c7fa3082e278f14b414, State: Initialized, Role: FOLLOWER
I20260929 20:47:25.512991 7903 consensus_queue.cc:260] T 0baa383f68dc4d4f9659115e503ae420 P dd36b520f02e4c7fa3082e278f14b414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a15205be8e418ba7b0856023a000ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33431 } } peers { permanent_uuid: "d343b75e87f142f29eb74970160c325e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41329 } } peers { permanent_uuid: "dd36b520f02e4c7fa3082e278f14b414" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39649 } }
I20260929 20:47:25.518085 7900 ts_tablet_manager.cc:1428] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee: Time spent starting tablet: real 0.011s user 0.003s sys 0.007s
I20260929 20:47:25.523931 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.518700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.97.2:50687)
0929 20:47:25.523527 (+ 4827us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.523672 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4706,"thread_start_us":4562,"threads_started":1}
I20260929 20:47:25.525803 7668 heartbeater.cc:503] Master 127.0.97.61:42803 was elected leader, sending a full tablet report...
I20260929 20:47:25.526087 7903 ts_tablet_manager.cc:1428] T 0baa383f68dc4d4f9659115e503ae420 P dd36b520f02e4c7fa3082e278f14b414: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260929 20:47:25.526980 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.524783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.97.3:60383)
0929 20:47:25.526616 (+ 1833us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.526749 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1717,"thread_start_us":1549,"threads_started":1}
I20260929 20:47:25.525811 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.525382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.97.2:46103)
0929 20:47:25.525574 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.525650 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 20:47:25.528118 7905 raft_consensus.cc:482] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:47:25.528402 7809 heartbeater.cc:503] Master 127.0.97.61:42803 was elected leader, sending a full tablet report...
I20260929 20:47:25.528420 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.528092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.97.3:51281)
0929 20:47:25.528215 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.528285 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 20:47:25.528369 7905 raft_consensus.cc:504] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a15205be8e418ba7b0856023a000ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33431 } } peers { permanent_uuid: "d343b75e87f142f29eb74970160c325e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41329 } } peers { permanent_uuid: "dd36b520f02e4c7fa3082e278f14b414" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39649 } }
I20260929 20:47:25.529477 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.511928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.97.1:60185)
0929 20:47:25.529083 (+ 17155us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.529216 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17044,"threads_started":1}
I20260929 20:47:25.530273 7905 leader_election.cc:288] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d343b75e87f142f29eb74970160c325e (127.0.97.1:41329), dd36b520f02e4c7fa3082e278f14b414 (127.0.97.3:39649)
I20260929 20:47:25.530678 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.530290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.97.1:44913)
0929 20:47:25.530474 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.530554 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 20:47:25.535593 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.530669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54223 (local address 127.0.97.1:41329)
0929 20:47:25.530845 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:25.530851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:25.530867 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:47:25.530923 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:25.530928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:25.530994 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:25.531117 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:25.531799 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.532475 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.534043 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.534783 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.534886 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:25.534975 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:25.535277 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 20:47:25.535393 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:47:25.535642 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.530382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41329 (local address 127.0.97.2:54223)
0929 20:47:25.530561 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.530575 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:25.530700 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:25.531318 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:25.531324 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:25.531351 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:25.531681 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:25.531697 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.532588 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.532594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:25.533904 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:25.533915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.534935 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.534941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:25.535145 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:25.535169 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:47:25.535245 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 20:47:25.535367 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 20:47:25.536360 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.530927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36059 (local address 127.0.97.3:39649)
0929 20:47:25.531082 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:25.531087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:25.531885 (+ 798us) server_negotiation.cc:407] Connection header received
0929 20:47:25.532022 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:25.532027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:25.532073 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:25.532140 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:25.532551 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.533189 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.534730 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.535488 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.535594 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:25.535682 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:25.536060 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 20:47:25.536183 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 20:47:25.536365 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.530668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39649 (local address 127.0.97.2:36059)
0929 20:47:25.531803 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.531816 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:25.531917 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:25.532159 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:25.532164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:25.532185 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:25.532431 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:25.532439 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.533319 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.533324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:25.534596 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:25.534607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.535775 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.535780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:25.535922 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:25.535952 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:47:25.536031 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 20:47:25.536109 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"thread_start_us":879,"threads_started":1}
I20260929 20:47:25.536373 7470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0baa383f68dc4d4f9659115e503ae420" candidate_uuid: "16a15205be8e418ba7b0856023a000ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d343b75e87f142f29eb74970160c325e" is_pre_election: true
I20260929 20:47:25.536561 7470 raft_consensus.cc:2454] T 0baa383f68dc4d4f9659115e503ae420 P d343b75e87f142f29eb74970160c325e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16a15205be8e418ba7b0856023a000ee in term 0.
I20260929 20:47:25.536989 7557 leader_election.cc:302] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16a15205be8e418ba7b0856023a000ee, d343b75e87f142f29eb74970160c325e; no voters:
I20260929 20:47:25.537359 7905 raft_consensus.cc:2781] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:47:25.537459 7905 raft_consensus.cc:482] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:47:25.537329 7762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0baa383f68dc4d4f9659115e503ae420" candidate_uuid: "16a15205be8e418ba7b0856023a000ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd36b520f02e4c7fa3082e278f14b414" is_pre_election: true
I20260929 20:47:25.537547 7905 raft_consensus.cc:3037] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:25.537665 7762 raft_consensus.cc:2454] T 0baa383f68dc4d4f9659115e503ae420 P dd36b520f02e4c7fa3082e278f14b414 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16a15205be8e418ba7b0856023a000ee in term 0.
I20260929 20:47:25.539819 7905 raft_consensus.cc:504] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a15205be8e418ba7b0856023a000ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33431 } } peers { permanent_uuid: "d343b75e87f142f29eb74970160c325e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41329 } } peers { permanent_uuid: "dd36b520f02e4c7fa3082e278f14b414" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39649 } }
I20260929 20:47:25.540333 7905 leader_election.cc:288] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee [CANDIDATE]: Term 1 election: Requested vote from peers d343b75e87f142f29eb74970160c325e (127.0.97.1:41329), dd36b520f02e4c7fa3082e278f14b414 (127.0.97.3:39649)
I20260929 20:47:25.540712 7470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0baa383f68dc4d4f9659115e503ae420" candidate_uuid: "16a15205be8e418ba7b0856023a000ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d343b75e87f142f29eb74970160c325e"
I20260929 20:47:25.540753 7762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0baa383f68dc4d4f9659115e503ae420" candidate_uuid: "16a15205be8e418ba7b0856023a000ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd36b520f02e4c7fa3082e278f14b414"
I20260929 20:47:25.540921 7762 raft_consensus.cc:3037] T 0baa383f68dc4d4f9659115e503ae420 P dd36b520f02e4c7fa3082e278f14b414 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:25.540938 7470 raft_consensus.cc:3037] T 0baa383f68dc4d4f9659115e503ae420 P d343b75e87f142f29eb74970160c325e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:25.543708 7470 raft_consensus.cc:2454] T 0baa383f68dc4d4f9659115e503ae420 P d343b75e87f142f29eb74970160c325e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16a15205be8e418ba7b0856023a000ee in term 1.
I20260929 20:47:25.543708 7762 raft_consensus.cc:2454] T 0baa383f68dc4d4f9659115e503ae420 P dd36b520f02e4c7fa3082e278f14b414 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16a15205be8e418ba7b0856023a000ee in term 1.
I20260929 20:47:25.544082 7557 leader_election.cc:302] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16a15205be8e418ba7b0856023a000ee, d343b75e87f142f29eb74970160c325e; no voters:
I20260929 20:47:25.544397 7905 raft_consensus.cc:2781] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:47:25.544795 7905 raft_consensus.cc:686] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee [term 1 LEADER]: Becoming Leader. State: Replica: 16a15205be8e418ba7b0856023a000ee, State: Running, Role: LEADER
I20260929 20:47:25.545156 7905 consensus_queue.cc:237] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a15205be8e418ba7b0856023a000ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33431 } } peers { permanent_uuid: "d343b75e87f142f29eb74970160c325e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41329 } } peers { permanent_uuid: "dd36b520f02e4c7fa3082e278f14b414" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39649 } }
I20260929 20:47:25.545616 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.545258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.97.2:60507)
0929 20:47:25.545421 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.545498 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 20:47:25.548286 7148 catalog_manager.cc:5495] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 16a15205be8e418ba7b0856023a000ee (127.0.97.2). New cstate: current_term: 1 leader_uuid: "16a15205be8e418ba7b0856023a000ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a15205be8e418ba7b0856023a000ee" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d343b75e87f142f29eb74970160c325e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd36b520f02e4c7fa3082e278f14b414" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39649 } health_report { overall_health: UNKNOWN } } }
I20260929 20:47:25.577263 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.572643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33682 (local address 127.0.97.2:33431)
0929 20:47:25.572807 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:25.572811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:25.572827 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:47:25.572971 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:25.572975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:25.573045 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:25.573158 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:47:25.573463 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.574161 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.575516 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.575906 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.575992 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:25.576353 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:47:25.576835 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:47:25.576934 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:25.577014 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 20:47:25.577071 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 20:47:25.580812 7470 raft_consensus.cc:1261] T 0baa383f68dc4d4f9659115e503ae420 P d343b75e87f142f29eb74970160c325e [term 1 FOLLOWER]: Refusing update from remote peer 16a15205be8e418ba7b0856023a000ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:47:25.581000 7762 raft_consensus.cc:1261] T 0baa383f68dc4d4f9659115e503ae420 P dd36b520f02e4c7fa3082e278f14b414 [term 1 FOLLOWER]: Refusing update from remote peer 16a15205be8e418ba7b0856023a000ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:47:25.581489 7913 consensus_queue.cc:1035] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d343b75e87f142f29eb74970160c325e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:47:25.581825 7905 consensus_queue.cc:1035] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd36b520f02e4c7fa3082e278f14b414" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:47:25.583142 7517 heartbeater.cc:503] Master 127.0.97.61:42803 was elected leader, sending a full tablet report...
I20260929 20:47:25.584115 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.583674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.97.1:44339)
0929 20:47:25.583823 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.583906 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 20:47:25.584913 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.583677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.97.3:59037)
0929 20:47:25.584625 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.584745 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":816}
I20260929 20:47:25.587990 7916 mvcc.cc:205] Tried to move back new op lower bound from 7334768007493181440 to 7334768007356637184. Current Snapshot: MvccSnapshot[applied={T|T < 7334768007493181440}]
I20260929 20:47:25.588534 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.588104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.97.2:45657)
0929 20:47:25.588277 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.588374 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 20:47:25.589294 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.588856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.97.3:41905)
0929 20:47:25.589054 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.589146 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 20:47:25.596622 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.596223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.97.2:35995)
0929 20:47:25.596379 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.596463 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 20:47:25.598532 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.598096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.97.1:49241)
0929 20:47:25.598293 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:25.598378 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 20:47:25.605216 7147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:47:25.605485 7147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:47:25.610581 7147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1097f58cae6a4662999174443b82b300: Sending DeleteTablet for 3 replicas of tablet 0baa383f68dc4d4f9659115e503ae420
I20260929 20:47:25.611480 7601 tablet_service.cc:1514] Processing DeleteTablet for tablet 0baa383f68dc4d4f9659115e503ae420 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:25 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:33680
I20260929 20:47:25.611667 7742 tablet_service.cc:1514] Processing DeleteTablet for tablet 0baa383f68dc4d4f9659115e503ae420 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:25 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:40390
I20260929 20:47:25.611699 7450 tablet_service.cc:1514] Processing DeleteTablet for tablet 0baa383f68dc4d4f9659115e503ae420 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:25 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:48128
I20260929 20:47:25.613109 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 7374
I20260929 20:47:25.623425 7927 tablet_replica.cc:330] T 0baa383f68dc4d4f9659115e503ae420 P d343b75e87f142f29eb74970160c325e: stopping tablet replica
I20260929 20:47:25.623639 7927 raft_consensus.cc:2229] T 0baa383f68dc4d4f9659115e503ae420 P d343b75e87f142f29eb74970160c325e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:47:25.623925 7927 raft_consensus.cc:2258] T 0baa383f68dc4d4f9659115e503ae420 P d343b75e87f142f29eb74970160c325e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:25.624325 7926 tablet_replica.cc:330] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee: stopping tablet replica
I20260929 20:47:25.624505 7926 raft_consensus.cc:2229] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:47:25.624898 7927 ts_tablet_manager.cc:1905] T 0baa383f68dc4d4f9659115e503ae420 P d343b75e87f142f29eb74970160c325e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:25.625399 7928 tablet_replica.cc:330] T 0baa383f68dc4d4f9659115e503ae420 P dd36b520f02e4c7fa3082e278f14b414: stopping tablet replica
I20260929 20:47:25.625573 7928 raft_consensus.cc:2229] T 0baa383f68dc4d4f9659115e503ae420 P dd36b520f02e4c7fa3082e278f14b414 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:47:25.625836 7928 raft_consensus.cc:2258] T 0baa383f68dc4d4f9659115e503ae420 P dd36b520f02e4c7fa3082e278f14b414 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:25.626077 7926 raft_consensus.cc:2258] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:25.627041 7928 ts_tablet_manager.cc:1905] T 0baa383f68dc4d4f9659115e503ae420 P dd36b520f02e4c7fa3082e278f14b414: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:25.630511 7927 ts_tablet_manager.cc:1918] T 0baa383f68dc4d4f9659115e503ae420 P d343b75e87f142f29eb74970160c325e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:47:25.630647 7927 log.cc:1196] T 0baa383f68dc4d4f9659115e503ae420 P d343b75e87f142f29eb74970160c325e: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/0baa383f68dc4d4f9659115e503ae420
I20260929 20:47:25.631138 7927 ts_tablet_manager.cc:1939] T 0baa383f68dc4d4f9659115e503ae420 P d343b75e87f142f29eb74970160c325e: Deleting consensus metadata
I20260929 20:47:25.632047 7135 catalog_manager.cc:4841] TS d343b75e87f142f29eb74970160c325e (127.0.97.1:41329): tablet 0baa383f68dc4d4f9659115e503ae420 (table test-table [id=ed13ad33ae9c4db3bb55ec31531dc241]) successfully deleted
I20260929 20:47:25.633736 7928 ts_tablet_manager.cc:1918] T 0baa383f68dc4d4f9659115e503ae420 P dd36b520f02e4c7fa3082e278f14b414: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:47:25.633862 7928 log.cc:1196] T 0baa383f68dc4d4f9659115e503ae420 P dd36b520f02e4c7fa3082e278f14b414: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/0baa383f68dc4d4f9659115e503ae420
I20260929 20:47:25.634269 7928 ts_tablet_manager.cc:1939] T 0baa383f68dc4d4f9659115e503ae420 P dd36b520f02e4c7fa3082e278f14b414: Deleting consensus metadata
I20260929 20:47:25.635147 7926 ts_tablet_manager.cc:1905] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:25.635298 7135 catalog_manager.cc:4841] TS dd36b520f02e4c7fa3082e278f14b414 (127.0.97.3:39649): tablet 0baa383f68dc4d4f9659115e503ae420 (table test-table [id=ed13ad33ae9c4db3bb55ec31531dc241]) successfully deleted
I20260929 20:47:25.640560 7926 ts_tablet_manager.cc:1918] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:47:25.640702 7926 log.cc:1196] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/0baa383f68dc4d4f9659115e503ae420
I20260929 20:47:25.641160 7926 ts_tablet_manager.cc:1939] T 0baa383f68dc4d4f9659115e503ae420 P 16a15205be8e418ba7b0856023a000ee: Deleting consensus metadata
I20260929 20:47:25.642364 7134 catalog_manager.cc:4841] TS 16a15205be8e418ba7b0856023a000ee (127.0.97.2:33431): tablet 0baa383f68dc4d4f9659115e503ae420 (table test-table [id=ed13ad33ae9c4db3bb55ec31531dc241]) successfully deleted
I20260929 20:47:25.645103 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.614265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48138 (local address 127.0.97.1:41329)
0929 20:47:25.614430 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:25.614437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:25.625560 (+ 11123us) server_negotiation.cc:407] Connection header received
0929 20:47:25.625748 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:25.625757 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:25.625828 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:25.625919 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:25.627584 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.628682 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.635111 (+ 6429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.635693 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.635821 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:25.638970 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:25.638988 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:25.639008 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:25.639090 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:25.642608 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:25.643361 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:25.643367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:25.643370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:25.643485 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:25.643858 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:25.643861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:25.643863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:25.644174 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:25.644395 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:25.644690 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 20:47:25.644858 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:47:25.645744 7510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:25.734855 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 7374
I20260929 20:47:25.755352 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 7530
I20260929 20:47:25.768292 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.756046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.0.97.2:33431)
0929 20:47:25.756227 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:25.756233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:25.756328 (+ 95us) server_negotiation.cc:407] Connection header received
0929 20:47:25.756502 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:25.756508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:25.756594 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:25.756687 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:25.757868 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.758614 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.759942 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.760390 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.760509 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:25.762834 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:25.762847 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:25.762863 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:25.762892 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:25.765695 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:25.766322 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:25.766330 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:25.766332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:25.766490 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:25.766828 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:25.766832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:25.766836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:25.767212 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:25.767518 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:25.767787 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 20:47:25.767983 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:47:25.768870 7661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:25.881342 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 7530
I20260929 20:47:25.896205 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 7679
I20260929 20:47:25.908573 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:25.896886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40398 (local address 127.0.97.3:39649)
0929 20:47:25.897049 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:25.897055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:25.897073 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:47:25.897276 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:25.897280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:25.897337 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:25.897416 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:25.898675 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.899425 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.900573 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:25.900974 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:25.901084 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:25.903412 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:25.903427 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:25.903444 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:25.903477 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:25.906076 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:25.906687 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:25.906694 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:25.906697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:25.906817 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:25.907188 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:25.907192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:25.907194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:25.907502 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:25.907769 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:25.908068 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 20:47:25.908275 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:47:25.909193 7802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:26.022317 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 7679
I20260929 20:47:26.025934 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:26.025325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.0.1:46826)
0929 20:47:26.025708 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:26.025788 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":196,"threads_started":1}
I20260929 20:47:26.027120 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:26.026784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45167 (local address 127.0.0.1:46830)
0929 20:47:26.026915 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:26.026980 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 20:47:26.039264 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 7010
I20260929 20:47:26.040215 7178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:26.156639 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 7010
W20260929 20:47:26.172492 7324 connection.cc:535] server connection from 127.0.0.1:38938 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 20:47:26.172854 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 7110
I20260929 20:47:26.173771 7367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:26.281972 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 7110
2026-09-29T20:47:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8706 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:47:26 dist-test-slave-blv7 krb5kdc[7939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:47:26 dist-test-slave-blv7 krb5kdc[7939](info): set up 2 sockets
Sep 29 20:47:26 dist-test-slave-blv7 krb5kdc[7939](info): commencing operation
krb5kdc: starting...
W20260929 20:47:27.275514 388 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:47:27 dist-test-slave-blv7 krb5kdc[7939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:47:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:47:27Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260929 20:47:27.466097 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35305
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:27.769207 7952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:27.769469 7952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:27.769582 7952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:27.776544 7952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:47:27.776646 7952 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 20:47:27.776691 7952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:27.776770 7952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:27.776830 7952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:47:27.776911 7952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:47:27.786558 7952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35305
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:27.787796 7952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:27.789047 7952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:27.799917 7957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:27.799853 7958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:27.799882 7960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:27.800663 7952 server_base.cc:1034] running on GCE node
I20260929 20:47:27.801440 7952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:27.802758 7952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:27.803963 7952 hybrid_clock.cc:648] HybridClock initialized: now 1790714847803883 us; error 58 us; skew 500 ppm
I20260929 20:47:27.804301 7952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:27 dist-test-slave-blv7 krb5kdc[7939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714847, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:27.816430 7952 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260929 20:47:27.817902 7952 webserver.cc:460] Webserver started at http://127.0.97.62:39803/ using document root <none> and password file <none>
I20260929 20:47:27.818585 7952 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:27.818681 7952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:27.818976 7952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:27.821807 7952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "a4466d3b70f94c64848db222c5423b10"
format_stamp: "Formatted at 2026-09-29 20:47:27 on dist-test-slave-blv7"
server_key: "d6361d6e10d52dd5bfbacf9b3c6b1f48"
server_key_iv: "99c6b5a6d3eea5edf7762f5e4aaa4ebb"
server_key_version: "encryptionkey@0"
I20260929 20:47:27.822566 7952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "a4466d3b70f94c64848db222c5423b10"
format_stamp: "Formatted at 2026-09-29 20:47:27 on dist-test-slave-blv7"
server_key: "d6361d6e10d52dd5bfbacf9b3c6b1f48"
server_key_iv: "99c6b5a6d3eea5edf7762f5e4aaa4ebb"
server_key_version: "encryptionkey@0"
I20260929 20:47:27.827455 7952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 20:47:27.830672 7967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:27.832069 7952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:47:27.832297 7952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "a4466d3b70f94c64848db222c5423b10"
format_stamp: "Formatted at 2026-09-29 20:47:27 on dist-test-slave-blv7"
server_key: "d6361d6e10d52dd5bfbacf9b3c6b1f48"
server_key_iv: "99c6b5a6d3eea5edf7762f5e4aaa4ebb"
server_key_version: "encryptionkey@0"
I20260929 20:47:27.832490 7952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:27.855813 7952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:27.859974 7952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:27.860216 7952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:27.880856 7952 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35305
I20260929 20:47:27.880883 8028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35305 every 8 connection(s)
I20260929 20:47:27.882055 7952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:47:27.885622 8029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:27.889976 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 7952
I20260929 20:47:27.890137 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:47:27.890719 388 external_mini_cluster.cc:1428] Setting key fc1c37443aff07ff9590e5b116413562
I20260929 20:47:27.895699 8029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10: Bootstrap starting.
I20260929 20:47:27.899214 8029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:27.900377 8029 log.cc:824] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:27.903517 8029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10: No bootstrap required, opened a new log
Sep 29 20:47:27 dist-test-slave-blv7 krb5kdc[7939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:47:27.912801 8032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:27.892499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55814 (local address 127.0.97.62:35305)
0929 20:47:27.893253 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:27.893272 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:27.893314 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:47:27.894922 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:27.894943 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:27.895363 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:27.895653 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:27.897283 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:27.898545 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:27.900210 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:27.901000 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:27.901181 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:27.905641 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:27.905650 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:27.905664 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:27.905705 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:27.908780 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:27.909612 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:27.909622 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:27.909628 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:27.909785 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:27.910186 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:27.910189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:27.910191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:27.910723 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:27.910934 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:27.911438 (+ 504us) server_negotiation.cc:299] Negotiation successful
0929 20:47:27.911780 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":305,"threads_started":1}
I20260929 20:47:27.912652 8029 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4466d3b70f94c64848db222c5423b10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35305 } }
I20260929 20:47:27.913012 8029 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:27.913127 8029 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4466d3b70f94c64848db222c5423b10, State: Initialized, Role: FOLLOWER
I20260929 20:47:27.913699 8029 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4466d3b70f94c64848db222c5423b10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35305 } }
I20260929 20:47:27.913926 8029 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:47:27.914073 8029 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:47:27.914228 8029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:27.916556 8029 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4466d3b70f94c64848db222c5423b10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35305 } }
I20260929 20:47:27.917063 8029 leader_election.cc:302] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4466d3b70f94c64848db222c5423b10; no voters:
I20260929 20:47:27.917618 8029 leader_election.cc:288] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:47:27.917724 8034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:47:27.918387 8034 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10 [term 1 LEADER]: Becoming Leader. State: Replica: a4466d3b70f94c64848db222c5423b10, State: Running, Role: LEADER
I20260929 20:47:27.918910 8034 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4466d3b70f94c64848db222c5423b10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35305 } }
I20260929 20:47:27.919582 8029 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:47:27.921468 8036 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4466d3b70f94c64848db222c5423b10. Latest consensus state: current_term: 1 leader_uuid: "a4466d3b70f94c64848db222c5423b10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4466d3b70f94c64848db222c5423b10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35305 } } }
I20260929 20:47:27.921905 8036 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10 [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:27.922230 8035 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4466d3b70f94c64848db222c5423b10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4466d3b70f94c64848db222c5423b10" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35305 } } }
I20260929 20:47:27.922534 8035 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10 [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:27.924099 8039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:47:27.928941 8039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:47:27.937258 8039 catalog_manager.cc:1347] Generated new cluster ID: a932a40ab6e442eca8f9ebfda21bc0de
I20260929 20:47:27.937350 8039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:47:27.945771 8039 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:47:27.947249 8039 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:47:27.955673 8039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10: Generated new TSK 0
I20260929 20:47:27.956763 8039 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260929 20:47:28.012560 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35305
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:28.324796 8055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:28.325098 8055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:28.325237 8055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:28.332434 8055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:28.332538 8055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:28.332662 8055 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:47:28.332768 8055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:47:28.342396 8055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:28.343677 8055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:28.344947 8055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:28.356841 8060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:28.356803 8063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:28.356846 8061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:28.357687 8055 server_base.cc:1034] running on GCE node
I20260929 20:47:28.358246 8055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:28.359102 8055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:28.360317 8055 hybrid_clock.cc:648] HybridClock initialized: now 1790714848360258 us; error 39 us; skew 500 ppm
I20260929 20:47:28.360585 8055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:28 dist-test-slave-blv7 krb5kdc[7939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:28.363451 8055 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 20:47:28.364722 8055 webserver.cc:460] Webserver started at http://127.0.97.1:39365/ using document root <none> and password file <none>
I20260929 20:47:28.365365 8055 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:28.365458 8055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:28.365732 8055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:28.368439 8055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "5086db73455d42d6ba8ecdae4169f287"
format_stamp: "Formatted at 2026-09-29 20:47:28 on dist-test-slave-blv7"
server_key: "df7732c2e07dc2f2ebd90ab72ff8294a"
server_key_iv: "07bb1273311382f219e3df0e5245249e"
server_key_version: "encryptionkey@0"
I20260929 20:47:28.369148 8055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5086db73455d42d6ba8ecdae4169f287"
format_stamp: "Formatted at 2026-09-29 20:47:28 on dist-test-slave-blv7"
server_key: "df7732c2e07dc2f2ebd90ab72ff8294a"
server_key_iv: "07bb1273311382f219e3df0e5245249e"
server_key_version: "encryptionkey@0"
I20260929 20:47:28.373893 8055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 20:47:28.377180 8070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:28.378440 8055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:47:28.378589 8055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "5086db73455d42d6ba8ecdae4169f287"
format_stamp: "Formatted at 2026-09-29 20:47:28 on dist-test-slave-blv7"
server_key: "df7732c2e07dc2f2ebd90ab72ff8294a"
server_key_iv: "07bb1273311382f219e3df0e5245249e"
server_key_version: "encryptionkey@0"
I20260929 20:47:28.378779 8055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:28.426896 8055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:28.430955 8055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:28.431227 8055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:28.433619 8055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:28.433702 8055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:28.433825 8055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:28.433879 8055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:47:28 dist-test-slave-blv7 krb5kdc[7939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:47:28.461326 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:28.434206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52811 (local address 127.0.97.62:35305)
0929 20:47:28.435374 (+ 1168us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:28.435381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:28.439915 (+ 4534us) server_negotiation.cc:407] Connection header received
0929 20:47:28.442910 (+ 2995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:28.442916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:28.443002 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:28.443143 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:28.445554 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:28.446694 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:28.449078 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:28.449620 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:28.449733 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:28.454300 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:28.454305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:28.454308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:28.454340 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:28.457390 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:28.458366 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:28.458371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:28.458375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:28.458508 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:28.459171 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:28.459177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:28.459180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:28.459543 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:28.459709 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:28.460681 (+ 972us) server_negotiation.cc:299] Negotiation successful
0929 20:47:28.460939 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1036,"thread_start_us":930,"threads_started":1}
I20260929 20:47:28.462354 8080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:28.434870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35305 (local address 127.0.97.1:52811)
0929 20:47:28.439716 (+ 4846us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:28.439778 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:28.442573 (+ 2795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:28.443584 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:28.443595 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:28.444483 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:28.445327 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:28.445355 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:28.446887 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:28.446892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:28.448922 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:28.448940 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:28.449840 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:28.449848 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:28.450152 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:28.450973 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:28.451002 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:28.454072 (+ 3070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:28.457572 (+ 3500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:28.457581 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:28.457607 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:28.458229 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:28.458644 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:28.458649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:28.458656 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:28.458987 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:28.459691 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:28.459702 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:28.460172 (+ 470us) client_negotiation.cc:769] Sending connection context
0929 20:47:28.460551 (+ 379us) client_negotiation.cc:240] Negotiation successful
0929 20:47:28.461191 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4529,"thread_start_us":4269,"threads_started":1}
I20260929 20:47:28.481377 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:28.469116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41369 (local address 127.0.97.62:35305)
0929 20:47:28.469290 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:28.469296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:28.469809 (+ 513us) server_negotiation.cc:407] Connection header received
0929 20:47:28.469928 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:28.469934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:28.470012 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:28.470120 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:28.470916 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:28.471950 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:28.473615 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:28.474224 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:28.474349 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:28.476441 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:28.476448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:28.476452 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:28.476487 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:28.478990 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:28.479622 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:28.479626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:28.479630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:28.479745 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:28.480047 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:28.480052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:28.480056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:28.480328 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:28.480447 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:28.480833 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 20:47:28.481029 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:47:28.481457 8166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:28.469116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35305 (local address 127.0.97.1:41369)
0929 20:47:28.469723 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:28.469741 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:28.469846 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:28.470127 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:28.470130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:28.470536 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:28.470801 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:28.470809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:28.472078 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:28.472082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:28.473471 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:28.473486 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:28.474391 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:28.474396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:28.474516 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:28.475156 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:28.475179 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:28.476206 (+ 1027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:28.479147 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:28.479152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:28.479155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:28.479521 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:28.479838 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:28.479843 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:28.479847 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:28.479961 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:28.480431 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:28.480435 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:28.480594 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 20:47:28.480716 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 20:47:28.481037 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":324,"threads_started":1}
I20260929 20:47:28.482833 8055 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38499
I20260929 20:47:28.482884 8190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38499 every 8 connection(s)
I20260929 20:47:28.484431 8055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:47:28.486279 8191 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35305
I20260929 20:47:28.486450 8191 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:28.486986 8191 heartbeater.cc:511] Master 127.0.97.62:35305 requested a full tablet report, sending...
I20260929 20:47:28.488885 7984 ts_manager.cc:194] Registered new tserver with Master: 5086db73455d42d6ba8ecdae4169f287 (127.0.97.1:38499)
I20260929 20:47:28.490384 7984 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52811
I20260929 20:47:28.494804 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 8055
I20260929 20:47:28.494979 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:47:28.495594 388 external_mini_cluster.cc:1428] Setting key f55d18e8ca57e8d8c1f3209d05d20360
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260929 20:47:28.535933 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35305
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:28.843125 8195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:28.843389 8195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:28.843518 8195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:28.850569 8195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:28.850672 8195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:28.850790 8195 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:47:28.850890 8195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:47:28.860332 8195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:28.861614 8195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:28.863121 8195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:28.876407 8201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:28.876461 8203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:28.876655 8200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:28.877368 8195 server_base.cc:1034] running on GCE node
I20260929 20:47:28.877965 8195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:28.878895 8195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:28.880141 8195 hybrid_clock.cc:648] HybridClock initialized: now 1790714848880024 us; error 54 us; skew 500 ppm
I20260929 20:47:28.880410 8195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:28 dist-test-slave-blv7 krb5kdc[7939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:28.883558 8195 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 20:47:28.884987 8195 webserver.cc:460] Webserver started at http://127.0.97.2:44597/ using document root <none> and password file <none>
I20260929 20:47:28.885622 8195 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:28.885715 8195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:28.885999 8195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:28.888774 8195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "f37db28cb1064e75ac5f7ec8c6d6f677"
format_stamp: "Formatted at 2026-09-29 20:47:28 on dist-test-slave-blv7"
server_key: "2b69f1b263480d069db0886a56cb5808"
server_key_iv: "80252e507ff7a61f29e20bc5e7f2239f"
server_key_version: "encryptionkey@0"
I20260929 20:47:28.889494 8195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f37db28cb1064e75ac5f7ec8c6d6f677"
format_stamp: "Formatted at 2026-09-29 20:47:28 on dist-test-slave-blv7"
server_key: "2b69f1b263480d069db0886a56cb5808"
server_key_iv: "80252e507ff7a61f29e20bc5e7f2239f"
server_key_version: "encryptionkey@0"
I20260929 20:47:28.894126 8195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260929 20:47:28.897219 8210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:28.898499 8195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:47:28.898638 8195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "f37db28cb1064e75ac5f7ec8c6d6f677"
format_stamp: "Formatted at 2026-09-29 20:47:28 on dist-test-slave-blv7"
server_key: "2b69f1b263480d069db0886a56cb5808"
server_key_iv: "80252e507ff7a61f29e20bc5e7f2239f"
server_key_version: "encryptionkey@0"
I20260929 20:47:28.898818 8195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:28.917563 8195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:28.921566 8195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:28.921813 8195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:28.924475 8195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:28.924577 8195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:28.924749 8195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:28.924851 8195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:47:28 dist-test-slave-blv7 krb5kdc[7939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714848, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:47:28.944903 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:28.924837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51751 (local address 127.0.97.62:35305)
0929 20:47:28.924976 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:28.924980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:28.926352 (+ 1372us) server_negotiation.cc:407] Connection header received
0929 20:47:28.928260 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:28.928266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:28.928346 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:28.928428 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:28.930520 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:28.931322 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:28.934125 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:28.934590 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:28.934714 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:28.938829 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:28.938834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:28.938837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:28.938866 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:28.941669 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:28.942638 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:28.942643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:28.942646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:28.942783 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:28.943280 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:28.943285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:28.943288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:28.943567 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:28.943697 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:28.944359 (+ 662us) server_negotiation.cc:299] Negotiation successful
0929 20:47:28.944593 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 20:47:28.945838 8219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:28.925433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35305 (local address 127.0.97.2:51751)
0929 20:47:28.926207 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:28.926253 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:28.928002 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:28.928681 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:28.928693 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:28.929423 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:28.930366 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:28.930390 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:28.931504 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:28.931510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:28.933980 (+ 2470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:28.933994 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:28.934828 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:28.934835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:28.935142 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:28.935839 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:28.935868 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:28.938620 (+ 2752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:28.941852 (+ 3232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:28.941862 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:28.941880 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:28.942518 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:28.942912 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:28.942917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:28.942920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:28.943177 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:28.943687 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:28.943694 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:28.944001 (+ 307us) client_negotiation.cc:769] Sending connection context
0929 20:47:28.944217 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 20:47:28.944682 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":304,"threads_started":1}
I20260929 20:47:28.964610 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:28.951392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35305 (local address 127.0.97.2:46987)
0929 20:47:28.951923 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:28.951936 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:28.952058 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:28.952483 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:28.952488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:28.953039 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:28.953495 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:28.953512 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:28.954684 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:28.954690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:28.955885 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:28.955896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:28.956679 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:28.956686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:28.956941 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:28.957654 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:28.957676 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:28.958658 (+ 982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:28.961831 (+ 3173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:28.961837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:28.961842 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:28.962453 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:28.962791 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:28.962795 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:28.962799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:28.962923 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:28.963671 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:28.963674 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:28.963873 (+ 199us) client_negotiation.cc:769] Sending connection context
0929 20:47:28.964034 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 20:47:28.964314 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":302,"threads_started":1}
I20260929 20:47:28.965270 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:28.951354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46987 (local address 127.0.97.62:35305)
0929 20:47:28.951493 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:28.951496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:28.952011 (+ 515us) server_negotiation.cc:407] Connection header received
0929 20:47:28.952161 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:28.952166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:28.952233 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:28.952319 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:28.953619 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:28.954546 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:28.956011 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:28.956465 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:28.956581 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:28.958843 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:28.958847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:28.958851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:28.958878 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:28.961670 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:28.962555 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:28.962560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:28.962563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:28.962701 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:28.963038 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:28.963042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:28.963045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:28.963487 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:28.963618 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:28.964874 (+ 1256us) server_negotiation.cc:299] Negotiation successful
0929 20:47:28.965053 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:47:28.977515 8195 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45097
I20260929 20:47:28.977562 8329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45097 every 8 connection(s)
I20260929 20:47:28.978643 8195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:47:28.980453 8330 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35305
I20260929 20:47:28.980638 8330 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:28.981211 8330 heartbeater.cc:511] Master 127.0.97.62:35305 requested a full tablet report, sending...
I20260929 20:47:28.982439 7984 ts_manager.cc:194] Registered new tserver with Master: f37db28cb1064e75ac5f7ec8c6d6f677 (127.0.97.2:45097)
I20260929 20:47:28.983299 7984 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51751
I20260929 20:47:28.983850 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 8195
I20260929 20:47:28.984014 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:47:28.984376 388 external_mini_cluster.cc:1428] Setting key 0143db984962272cb79aa2407ce17222
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 20:47:29.025939 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35305
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:29.327387 8334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:29.327658 8334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:29.327795 8334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:29.334740 8334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:29.334854 8334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:29.334982 8334 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:47:29.335112 8334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:47:29.344517 8334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:29.345721 8334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:29.347002 8334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:29.359406 8342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:29.359437 8340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:29.359828 8339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:29.360159 8334 server_base.cc:1034] running on GCE node
I20260929 20:47:29.360689 8334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:29.361526 8334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:29.362730 8334 hybrid_clock.cc:648] HybridClock initialized: now 1790714849362668 us; error 30 us; skew 500 ppm
I20260929 20:47:29.362990 8334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:29 dist-test-slave-blv7 krb5kdc[7939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714849, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:29.365877 8334 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 20:47:29.367177 8334 webserver.cc:460] Webserver started at http://127.0.97.3:40487/ using document root <none> and password file <none>
I20260929 20:47:29.367792 8334 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:29.367884 8334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:29.368161 8334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:29.370813 8334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "2fb8841de3304be998e952b840b3bc7a"
format_stamp: "Formatted at 2026-09-29 20:47:29 on dist-test-slave-blv7"
server_key: "7c7f51dd7c4558e0800b607a90df86a7"
server_key_iv: "a44c17c640b851a4f7ecc2828b7f9dab"
server_key_version: "encryptionkey@0"
I20260929 20:47:29.371577 8334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2fb8841de3304be998e952b840b3bc7a"
format_stamp: "Formatted at 2026-09-29 20:47:29 on dist-test-slave-blv7"
server_key: "7c7f51dd7c4558e0800b607a90df86a7"
server_key_iv: "a44c17c640b851a4f7ecc2828b7f9dab"
server_key_version: "encryptionkey@0"
I20260929 20:47:29.376215 8334 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 20:47:29.379379 8349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:29.380556 8334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260929 20:47:29.380686 8334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "2fb8841de3304be998e952b840b3bc7a"
format_stamp: "Formatted at 2026-09-29 20:47:29 on dist-test-slave-blv7"
server_key: "7c7f51dd7c4558e0800b607a90df86a7"
server_key_iv: "a44c17c640b851a4f7ecc2828b7f9dab"
server_key_version: "encryptionkey@0"
I20260929 20:47:29.380888 8334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:29.397061 8334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:29.400830 8334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:29.401046 8334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:29.403790 8334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:29.403867 8334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:29.403982 8334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:29.404039 8334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:47:29 dist-test-slave-blv7 krb5kdc[7939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714849, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:47:29.426270 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:29.404268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43369 (local address 127.0.97.62:35305)
0929 20:47:29.404445 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:29.404451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:29.405803 (+ 1352us) server_negotiation.cc:407] Connection header received
0929 20:47:29.408494 (+ 2691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:29.408499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:29.408580 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:29.408683 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:29.410913 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.411840 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.414582 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.415139 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.415270 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:29.420001 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:29.420006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:29.420010 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:29.420042 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:29.422753 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:29.423608 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:29.423615 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:29.423618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:29.423811 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:29.424251 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:29.424256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:29.424260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:29.424656 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:29.424834 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:29.425869 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0929 20:47:29.426028 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:47:29.428059 8358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:29.404807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35305 (local address 127.0.97.3:43369)
0929 20:47:29.405606 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:29.405669 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:29.408178 (+ 2509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:29.408933 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:29.408943 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:29.409767 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:29.410703 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:29.410732 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.412028 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.412035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:29.414431 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:29.414446 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.415385 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.415393 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:29.415693 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:29.416514 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:29.416549 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:29.419760 (+ 3211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:29.422877 (+ 3117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:29.422886 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:29.422905 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:29.423479 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:29.423922 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:29.423926 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:29.423929 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:29.424145 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:29.424913 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:29.424922 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:29.425408 (+ 486us) client_negotiation.cc:769] Sending connection context
0929 20:47:29.425769 (+ 361us) client_negotiation.cc:240] Negotiation successful
0929 20:47:29.426352 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":300,"threads_started":1}
I20260929 20:47:29.453197 8334 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44777
I20260929 20:47:29.453236 8468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44777 every 8 connection(s)
I20260929 20:47:29.454609 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:29.439977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35305 (local address 127.0.97.3:55245)
0929 20:47:29.440591 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:29.440610 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:29.440781 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:29.441354 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:29.441360 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:29.442051 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:29.442554 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:29.442570 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.444001 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.444005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:29.445713 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:29.445734 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.446798 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.446803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:29.446911 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:29.447641 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:29.447658 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:29.448452 (+ 794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:29.451961 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:29.451966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:29.451977 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:29.452520 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:29.452937 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:29.452942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:29.452946 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:29.453104 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:29.453753 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:29.453757 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:29.453917 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 20:47:29.454037 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 20:47:29.454301 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":322,"threads_started":1}
I20260929 20:47:29.454713 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:29.439952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55245 (local address 127.0.97.62:35305)
0929 20:47:29.440128 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:29.440143 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:29.440724 (+ 581us) server_negotiation.cc:407] Connection header received
0929 20:47:29.440920 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:29.440926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:29.441011 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:29.441141 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:29.442771 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.443797 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.445873 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.446603 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.446739 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:29.448644 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:29.448651 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:29.448657 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:29.448695 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:29.451824 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:29.452624 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:29.452630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:29.452634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:29.452813 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:29.453271 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:29.453276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:29.453278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:29.453599 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:29.453740 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:29.454170 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 20:47:29.454383 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:47:29.454771 8334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:47:29.457796 8469 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35305
I20260929 20:47:29.458034 8469 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:29.458674 8469 heartbeater.cc:511] Master 127.0.97.62:35305 requested a full tablet report, sending...
I20260929 20:47:29.459899 7984 ts_manager.cc:194] Registered new tserver with Master: 2fb8841de3304be998e952b840b3bc7a (127.0.97.3:44777)
I20260929 20:47:29.460666 7984 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43369
I20260929 20:47:29.463255 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 8334
I20260929 20:47:29.463399 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:47:29.463796 388 external_mini_cluster.cc:1428] Setting key 56557bf7566f72caaa214a50baf5ac8d
I20260929 20:47:29.466481 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:47:29.493000 8191 heartbeater.cc:503] Master 127.0.97.62:35305 was elected leader, sending a full tablet report...
Sep 29 20:47:29 dist-test-slave-blv7 krb5kdc[7939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714849, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:47:29 dist-test-slave-blv7 krb5kdc[7939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:29 dist-test-slave-blv7 krb5kdc[7939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 20:47:29.511399 8079 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:29.501937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55826 (local address 127.0.97.62:35305)
0929 20:47:29.502084 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:29.502088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:29.502403 (+ 315us) server_negotiation.cc:407] Connection header received
0929 20:47:29.502563 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:29.502566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:29.502625 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:29.502701 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:29.503956 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.504663 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.505922 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.506317 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.506405 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:29.511097 (+ 4692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55826: BlockingRecv error: recv got EOF from 127.0.0.1:55826 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 20:47:29 dist-test-slave-blv7 krb5kdc[7939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714849, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:47:29.526255 8079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:29.513932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55838 (local address 127.0.97.62:35305)
0929 20:47:29.514125 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:29.514130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:29.514363 (+ 233us) server_negotiation.cc:407] Connection header received
0929 20:47:29.514520 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:29.514524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:29.514590 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:29.514672 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:29.515794 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.516521 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.517679 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.518077 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.518181 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:29.521609 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:29.521614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:29.521617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:29.521648 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:29.523880 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:29.524480 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:29.524485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:29.524487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:29.524601 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:29.524924 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:29.524929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:29.524932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:29.525216 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:29.525473 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:29.525728 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 20:47:29.525952 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:47:29.529933 7984 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:47:29.536491 7984 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:47:29.568104 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:29.547726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45097 (local address 127.0.0.1:52660)
0929 20:47:29.559596 (+ 11870us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:29.559620 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:29.559799 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:29.560931 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:29.560935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:29.560962 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:29.561408 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:29.561424 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.563194 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.563199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:29.565531 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:29.565542 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.567103 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.567109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:29.567268 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:29.567341 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 20:47:29.567471 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 20:47:29.567651 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11690,"mutex_wait_us":74,"thread_start_us":239,"threads_started":1}
I20260929 20:47:29.568260 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:29.547958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52660 (local address 127.0.97.2:45097)
0929 20:47:29.548928 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:29.548934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:29.559760 (+ 10826us) server_negotiation.cc:407] Connection header received
0929 20:47:29.559966 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:29.559976 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:29.560309 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:29.560524 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:29.561593 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.563053 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.565669 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.566893 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.567048 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:29.567247 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:29.567512 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 20:47:29.567994 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":829,"thread_start_us":702,"threads_started":1}
I20260929 20:47:29.568631 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:29.547639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48286 (local address 127.0.97.1:38499)
0929 20:47:29.560264 (+ 12625us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:29.560285 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:29.560313 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:47:29.560403 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:29.560411 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:29.560640 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:29.560832 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:29.562304 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.563531 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.565986 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.567377 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.567487 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:29.567676 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:29.568221 (+ 545us) server_negotiation.cc:299] Negotiation successful
0929 20:47:29.568378 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12488,"thread_start_us":12373,"threads_started":1}
I20260929 20:47:29.568634 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:29.549507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48414 (local address 127.0.97.3:44777)
0929 20:47:29.561395 (+ 11888us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:29.561401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:29.561425 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:47:29.561493 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:29.561497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:29.561678 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:29.561847 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:29.562494 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.563800 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.566186 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.567144 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.567252 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:29.567473 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:29.568191 (+ 718us) server_negotiation.cc:299] Negotiation successful
0929 20:47:29.568355 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11751,"thread_start_us":12167,"threads_started":1}
I20260929 20:47:29.569026 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:29.549325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44777 (local address 127.0.0.1:48414)
0929 20:47:29.560546 (+ 11221us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:29.560562 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:29.560663 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:29.561915 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:29.561919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:29.561945 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:29.562328 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:29.562337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.563929 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.563934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:29.566051 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:29.566059 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.567600 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.567606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:29.567750 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:29.567787 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 20:47:29.567871 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 20:47:29.567980 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11118,"mutex_wait_us":2,"thread_start_us":11372,"threads_started":1}
I20260929 20:47:29.569367 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:29.547502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38499 (local address 127.0.0.1:48286)
0929 20:47:29.548319 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:29.548362 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:29.548587 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:29.560852 (+ 12265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:29.560859 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:29.560892 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:29.562114 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:29.562126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.563643 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.563648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:29.565846 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:29.565859 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.567840 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.567847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:29.568009 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:29.568043 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:47:29.568802 (+ 759us) client_negotiation.cc:240] Negotiation successful
0929 20:47:29.568932 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":413,"thread_start_us":13261,"threads_started":1}
I20260929 20:47:29.571975 8260 tablet_service.cc:1467] Processing CreateTablet for tablet c9d5bc7b8b5b4cd991cfea64eba09a9d (DEFAULT_TABLE table=test-table [id=97a63517256e435fb9d7c494358d59c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:29.572031 8399 tablet_service.cc:1467] Processing CreateTablet for tablet c9d5bc7b8b5b4cd991cfea64eba09a9d (DEFAULT_TABLE table=test-table [id=97a63517256e435fb9d7c494358d59c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:29.573205 8399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d5bc7b8b5b4cd991cfea64eba09a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:29.573691 8260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d5bc7b8b5b4cd991cfea64eba09a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:29.574744 8121 tablet_service.cc:1467] Processing CreateTablet for tablet c9d5bc7b8b5b4cd991cfea64eba09a9d (DEFAULT_TABLE table=test-table [id=97a63517256e435fb9d7c494358d59c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:29.575955 8121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9d5bc7b8b5b4cd991cfea64eba09a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:29.584649 8493 tablet_bootstrap.cc:492] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 2fb8841de3304be998e952b840b3bc7a: Bootstrap starting.
I20260929 20:47:29.587738 8493 tablet_bootstrap.cc:654] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 2fb8841de3304be998e952b840b3bc7a: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:29.588359 8494 tablet_bootstrap.cc:492] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677: Bootstrap starting.
I20260929 20:47:29.589072 8493 log.cc:824] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 2fb8841de3304be998e952b840b3bc7a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:29.590412 8495 tablet_bootstrap.cc:492] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 5086db73455d42d6ba8ecdae4169f287: Bootstrap starting.
I20260929 20:47:29.591869 8494 tablet_bootstrap.cc:654] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:29.591955 8493 tablet_bootstrap.cc:492] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 2fb8841de3304be998e952b840b3bc7a: No bootstrap required, opened a new log
I20260929 20:47:29.592188 8493 ts_tablet_manager.cc:1397] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 2fb8841de3304be998e952b840b3bc7a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 20:47:29.593086 8494 log.cc:824] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:29.593669 8495 tablet_bootstrap.cc:654] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 5086db73455d42d6ba8ecdae4169f287: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:29.594797 8495 log.cc:824] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 5086db73455d42d6ba8ecdae4169f287: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:29.596474 8494 tablet_bootstrap.cc:492] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677: No bootstrap required, opened a new log
I20260929 20:47:29.596725 8494 ts_tablet_manager.cc:1397] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 20:47:29.597522 8495 tablet_bootstrap.cc:492] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 5086db73455d42d6ba8ecdae4169f287: No bootstrap required, opened a new log
I20260929 20:47:29.597761 8495 ts_tablet_manager.cc:1397] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 5086db73455d42d6ba8ecdae4169f287: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 20:47:29.600359 8493 raft_consensus.cc:348] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:29.600689 8493 raft_consensus.cc:374] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:29.600822 8493 raft_consensus.cc:729] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb8841de3304be998e952b840b3bc7a, State: Initialized, Role: FOLLOWER
I20260929 20:47:29.601378 8493 consensus_queue.cc:260] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 2fb8841de3304be998e952b840b3bc7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:29.602531 8469 heartbeater.cc:503] Master 127.0.97.62:35305 was elected leader, sending a full tablet report...
I20260929 20:47:29.602941 8493 ts_tablet_manager.cc:1428] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 2fb8841de3304be998e952b840b3bc7a: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 20:47:29.607362 8495 raft_consensus.cc:348] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:29.607748 8495 raft_consensus.cc:374] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:29.607834 8495 raft_consensus.cc:729] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5086db73455d42d6ba8ecdae4169f287, State: Initialized, Role: FOLLOWER
I20260929 20:47:29.608390 8495 consensus_queue.cc:260] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 5086db73455d42d6ba8ecdae4169f287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:29.608551 8494 raft_consensus.cc:348] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:29.608948 8494 raft_consensus.cc:374] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:29.609040 8494 raft_consensus.cc:729] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f37db28cb1064e75ac5f7ec8c6d6f677, State: Initialized, Role: FOLLOWER
I20260929 20:47:29.609577 8494 consensus_queue.cc:260] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:29.609997 8495 ts_tablet_manager.cc:1428] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 5086db73455d42d6ba8ecdae4169f287: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260929 20:47:29.610603 8330 heartbeater.cc:503] Master 127.0.97.62:35305 was elected leader, sending a full tablet report...
I20260929 20:47:29.611135 8494 ts_tablet_manager.cc:1428] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260929 20:47:29.699914 8501 raft_consensus.cc:482] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:47:29.700251 8501 raft_consensus.cc:504] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:29.702119 8501 leader_election.cc:288] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5086db73455d42d6ba8ecdae4169f287 (127.0.97.1:38499), 2fb8841de3304be998e952b840b3bc7a (127.0.97.3:44777)
W20260929 20:47:29.706370 8470 tablet.cc:2369] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 2fb8841de3304be998e952b840b3bc7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:47:29.708451 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:29.702769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49353 (local address 127.0.97.3:44777)
0929 20:47:29.702952 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:29.702958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:29.703301 (+ 343us) server_negotiation.cc:407] Connection header received
0929 20:47:29.703516 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:29.703520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:29.703576 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:29.703655 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:29.704144 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.704876 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.706835 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.707628 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.707754 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:29.707858 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:29.708110 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 20:47:29.708229 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 20:47:29.708635 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:29.702401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44777 (local address 127.0.97.2:49353)
0929 20:47:29.703200 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:29.703223 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:29.703396 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:29.703669 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:29.703673 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:29.703694 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:29.704023 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:29.704033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.704985 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.704990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:29.706684 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:29.706696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.707829 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.707836 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:29.707975 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:29.707998 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:47:29.708076 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 20:47:29.708198 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":1,"spinlock_wait_cycles":768,"thread_start_us":553,"threads_started":1}
I20260929 20:47:29.709177 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:29.702568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60061 (local address 127.0.97.1:38499)
0929 20:47:29.702912 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:29.702919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:29.703470 (+ 551us) server_negotiation.cc:407] Connection header received
0929 20:47:29.703658 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:29.703664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:29.703735 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:29.703816 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:29.704410 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.705106 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.707386 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.708204 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.708333 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:29.708439 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:29.708839 (+ 400us) server_negotiation.cc:299] Negotiation successful
0929 20:47:29.708957 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260929 20:47:29.709246 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:29.702498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38499 (local address 127.0.97.2:60061)
0929 20:47:29.703375 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:29.703390 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:29.703516 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:29.703851 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:29.703856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:29.703879 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:29.704242 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:29.704255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.705249 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.705255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:29.707229 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:29.707244 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.708427 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.708435 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:29.708641 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:29.708675 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:47:29.708811 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 20:47:29.708932 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":30,"thread_start_us":541,"threads_started":1}
I20260929 20:47:29.709730 8419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d5bc7b8b5b4cd991cfea64eba09a9d" candidate_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fb8841de3304be998e952b840b3bc7a" is_pre_election: true
I20260929 20:47:29.710035 8141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d5bc7b8b5b4cd991cfea64eba09a9d" candidate_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5086db73455d42d6ba8ecdae4169f287" is_pre_election: true
I20260929 20:47:29.710212 8419 raft_consensus.cc:2454] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f37db28cb1064e75ac5f7ec8c6d6f677 in term 0.
I20260929 20:47:29.710461 8141 raft_consensus.cc:2454] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f37db28cb1064e75ac5f7ec8c6d6f677 in term 0.
I20260929 20:47:29.710872 8214 leader_election.cc:302] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb8841de3304be998e952b840b3bc7a, f37db28cb1064e75ac5f7ec8c6d6f677; no voters:
I20260929 20:47:29.711256 8501 raft_consensus.cc:2781] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:47:29.711395 8501 raft_consensus.cc:482] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:47:29.711494 8501 raft_consensus.cc:3037] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:29.713804 8501 raft_consensus.cc:504] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:29.714418 8501 leader_election.cc:288] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677 [CANDIDATE]: Term 1 election: Requested vote from peers 5086db73455d42d6ba8ecdae4169f287 (127.0.97.1:38499), 2fb8841de3304be998e952b840b3bc7a (127.0.97.3:44777)
I20260929 20:47:29.714740 8141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d5bc7b8b5b4cd991cfea64eba09a9d" candidate_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5086db73455d42d6ba8ecdae4169f287"
I20260929 20:47:29.714910 8141 raft_consensus.cc:3037] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:29.714856 8419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9d5bc7b8b5b4cd991cfea64eba09a9d" candidate_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fb8841de3304be998e952b840b3bc7a"
I20260929 20:47:29.715097 8419 raft_consensus.cc:3037] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:29.717031 8141 raft_consensus.cc:2454] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 5086db73455d42d6ba8ecdae4169f287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f37db28cb1064e75ac5f7ec8c6d6f677 in term 1.
I20260929 20:47:29.717442 8419 raft_consensus.cc:2454] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 2fb8841de3304be998e952b840b3bc7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f37db28cb1064e75ac5f7ec8c6d6f677 in term 1.
I20260929 20:47:29.717423 8212 leader_election.cc:302] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5086db73455d42d6ba8ecdae4169f287, f37db28cb1064e75ac5f7ec8c6d6f677; no voters:
I20260929 20:47:29.717749 8501 raft_consensus.cc:2781] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:47:29.718243 8501 raft_consensus.cc:686] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677 [term 1 LEADER]: Becoming Leader. State: Replica: f37db28cb1064e75ac5f7ec8c6d6f677, State: Running, Role: LEADER
I20260929 20:47:29.718714 8501 consensus_queue.cc:237] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:29.723075 7983 catalog_manager.cc:5495] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677 reported cstate change: term changed from 0 to 1, leader changed from <none> to f37db28cb1064e75ac5f7ec8c6d6f677 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } health_report { overall_health: UNKNOWN } } }
W20260929 20:47:29.730376 8331 tablet.cc:2369] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:47:29.736604 8192 tablet.cc:2369] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 5086db73455d42d6ba8ecdae4169f287: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:47:29.766337 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:29.761254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52664 (local address 127.0.97.2:45097)
0929 20:47:29.761440 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:29.761446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:29.761530 (+ 84us) server_negotiation.cc:407] Connection header received
0929 20:47:29.761792 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:29.761796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:29.761876 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:29.761999 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:47:29.762331 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.763062 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.764348 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.764777 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.764890 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:29.765260 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:47:29.765778 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:47:29.765901 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:29.765993 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 20:47:29.766070 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:47:29.773732 8419 raft_consensus.cc:1261] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 2fb8841de3304be998e952b840b3bc7a [term 1 FOLLOWER]: Refusing update from remote peer f37db28cb1064e75ac5f7ec8c6d6f677: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:47:29.773764 8141 raft_consensus.cc:1261] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 5086db73455d42d6ba8ecdae4169f287 [term 1 FOLLOWER]: Refusing update from remote peer f37db28cb1064e75ac5f7ec8c6d6f677: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:47:29.774549 8504 consensus_queue.cc:1035] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:47:29.774879 8501 consensus_queue.cc:1035] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:47:29.784956 8507 mvcc.cc:205] Tried to move back new op lower bound from 7334768024664858624 to 7334768024451993600. Current Snapshot: MvccSnapshot[applied={T|T < 7334768024664858624}]
I20260929 20:47:29.785633 8508 mvcc.cc:205] Tried to move back new op lower bound from 7334768024664858624 to 7334768024451993600. Current Snapshot: MvccSnapshot[applied={T|T < 7334768024664858624}]
I20260929 20:47:29.790308 8509 mvcc.cc:205] Tried to move back new op lower bound from 7334768024664858624 to 7334768024451993600. Current Snapshot: MvccSnapshot[applied={T|T < 7334768024664858624}]
I20260929 20:47:29.797469 7984 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:47:29.797713 7984 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:47:29.800990 7984 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10: Sending DeleteTablet for 3 replicas of tablet c9d5bc7b8b5b4cd991cfea64eba09a9d
I20260929 20:47:29.802110 8121 tablet_service.cc:1514] Processing DeleteTablet for tablet c9d5bc7b8b5b4cd991cfea64eba09a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:29 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48286
I20260929 20:47:29.802213 8260 tablet_service.cc:1514] Processing DeleteTablet for tablet c9d5bc7b8b5b4cd991cfea64eba09a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:29 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52660
I20260929 20:47:29.803076 8399 tablet_service.cc:1514] Processing DeleteTablet for tablet c9d5bc7b8b5b4cd991cfea64eba09a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:29 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48414
I20260929 20:47:29.803860 8516 tablet_replica.cc:330] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 5086db73455d42d6ba8ecdae4169f287: stopping tablet replica
I20260929 20:47:29.803889 7984 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:47:29.804500 7984 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:47:29.804504 8516 raft_consensus.cc:2229] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 5086db73455d42d6ba8ecdae4169f287 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:47:29.805078 8516 raft_consensus.cc:2258] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 5086db73455d42d6ba8ecdae4169f287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:29.806000 8517 tablet_replica.cc:330] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677: stopping tablet replica
I20260929 20:47:29.806458 8517 raft_consensus.cc:2229] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:47:29.807175 8517 raft_consensus.cc:2258] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:29.807298 8516 ts_tablet_manager.cc:1905] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 5086db73455d42d6ba8ecdae4169f287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:29.808089 8518 tablet_replica.cc:330] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 2fb8841de3304be998e952b840b3bc7a: stopping tablet replica
I20260929 20:47:29.808456 8518 raft_consensus.cc:2229] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 2fb8841de3304be998e952b840b3bc7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:47:29.808935 8518 raft_consensus.cc:2258] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 2fb8841de3304be998e952b840b3bc7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:29.810822 8518 ts_tablet_manager.cc:1905] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 2fb8841de3304be998e952b840b3bc7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:29.811796 8517 ts_tablet_manager.cc:1905] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:29.814314 8516 ts_tablet_manager.cc:1918] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 5086db73455d42d6ba8ecdae4169f287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:47:29.814466 8516 log.cc:1196] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 5086db73455d42d6ba8ecdae4169f287: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/c9d5bc7b8b5b4cd991cfea64eba09a9d
I20260929 20:47:29.814958 8516 ts_tablet_manager.cc:1939] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 5086db73455d42d6ba8ecdae4169f287: Deleting consensus metadata
I20260929 20:47:29.814926 8121 tablet_service.cc:1467] Processing CreateTablet for tablet bba0feb2d6f94aab92d201273c33f713 (DEFAULT_TABLE table=test-table [id=5876ba51ddec411388f36d67f8b5839a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:29.815392 8121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bba0feb2d6f94aab92d201273c33f713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:29.816772 7969 catalog_manager.cc:4841] TS 5086db73455d42d6ba8ecdae4169f287 (127.0.97.1:38499): tablet c9d5bc7b8b5b4cd991cfea64eba09a9d (table test-table [id=97a63517256e435fb9d7c494358d59c3]) successfully deleted
I20260929 20:47:29.817332 8260 tablet_service.cc:1467] Processing CreateTablet for tablet bba0feb2d6f94aab92d201273c33f713 (DEFAULT_TABLE table=test-table [id=5876ba51ddec411388f36d67f8b5839a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:29.817776 8260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bba0feb2d6f94aab92d201273c33f713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:29.820442 8517 ts_tablet_manager.cc:1918] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:47:29.820617 8517 log.cc:1196] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/c9d5bc7b8b5b4cd991cfea64eba09a9d
I20260929 20:47:29.820840 8399 tablet_service.cc:1467] Processing CreateTablet for tablet bba0feb2d6f94aab92d201273c33f713 (DEFAULT_TABLE table=test-table [id=5876ba51ddec411388f36d67f8b5839a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:29.821135 8517 ts_tablet_manager.cc:1939] T c9d5bc7b8b5b4cd991cfea64eba09a9d P f37db28cb1064e75ac5f7ec8c6d6f677: Deleting consensus metadata
I20260929 20:47:29.821151 8518 ts_tablet_manager.cc:1918] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 2fb8841de3304be998e952b840b3bc7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:47:29.821291 8399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bba0feb2d6f94aab92d201273c33f713. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:29.821334 8518 log.cc:1196] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 2fb8841de3304be998e952b840b3bc7a: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/c9d5bc7b8b5b4cd991cfea64eba09a9d
I20260929 20:47:29.821807 8518 ts_tablet_manager.cc:1939] T c9d5bc7b8b5b4cd991cfea64eba09a9d P 2fb8841de3304be998e952b840b3bc7a: Deleting consensus metadata
I20260929 20:47:29.822829 7970 catalog_manager.cc:4841] TS f37db28cb1064e75ac5f7ec8c6d6f677 (127.0.97.2:45097): tablet c9d5bc7b8b5b4cd991cfea64eba09a9d (table test-table [id=97a63517256e435fb9d7c494358d59c3]) successfully deleted
I20260929 20:47:29.823487 7971 catalog_manager.cc:4841] TS 2fb8841de3304be998e952b840b3bc7a (127.0.97.3:44777): tablet c9d5bc7b8b5b4cd991cfea64eba09a9d (table test-table [id=97a63517256e435fb9d7c494358d59c3]) successfully deleted
I20260929 20:47:29.827791 8495 tablet_bootstrap.cc:492] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287: Bootstrap starting.
I20260929 20:47:29.828336 8494 tablet_bootstrap.cc:492] T bba0feb2d6f94aab92d201273c33f713 P f37db28cb1064e75ac5f7ec8c6d6f677: Bootstrap starting.
I20260929 20:47:29.829394 8495 tablet_bootstrap.cc:654] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:29.829933 8494 tablet_bootstrap.cc:654] T bba0feb2d6f94aab92d201273c33f713 P f37db28cb1064e75ac5f7ec8c6d6f677: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:29.830617 8493 tablet_bootstrap.cc:492] T bba0feb2d6f94aab92d201273c33f713 P 2fb8841de3304be998e952b840b3bc7a: Bootstrap starting.
I20260929 20:47:29.832190 8495 tablet_bootstrap.cc:492] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287: No bootstrap required, opened a new log
I20260929 20:47:29.832193 8494 tablet_bootstrap.cc:492] T bba0feb2d6f94aab92d201273c33f713 P f37db28cb1064e75ac5f7ec8c6d6f677: No bootstrap required, opened a new log
I20260929 20:47:29.832194 8493 tablet_bootstrap.cc:654] T bba0feb2d6f94aab92d201273c33f713 P 2fb8841de3304be998e952b840b3bc7a: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:29.832330 8495 ts_tablet_manager.cc:1397] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260929 20:47:29.832337 8494 ts_tablet_manager.cc:1397] T bba0feb2d6f94aab92d201273c33f713 P f37db28cb1064e75ac5f7ec8c6d6f677: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260929 20:47:29.833071 8495 raft_consensus.cc:348] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:29.833292 8495 raft_consensus.cc:374] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:29.833386 8495 raft_consensus.cc:729] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5086db73455d42d6ba8ecdae4169f287, State: Initialized, Role: FOLLOWER
I20260929 20:47:29.833631 8495 consensus_queue.cc:260] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:29.833709 8494 raft_consensus.cc:348] T bba0feb2d6f94aab92d201273c33f713 P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:29.833981 8494 raft_consensus.cc:374] T bba0feb2d6f94aab92d201273c33f713 P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:29.834080 8494 raft_consensus.cc:729] T bba0feb2d6f94aab92d201273c33f713 P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f37db28cb1064e75ac5f7ec8c6d6f677, State: Initialized, Role: FOLLOWER
I20260929 20:47:29.834197 8495 ts_tablet_manager.cc:1428] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 20:47:29.834399 8494 consensus_queue.cc:260] T bba0feb2d6f94aab92d201273c33f713 P f37db28cb1064e75ac5f7ec8c6d6f677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:29.835796 8493 tablet_bootstrap.cc:492] T bba0feb2d6f94aab92d201273c33f713 P 2fb8841de3304be998e952b840b3bc7a: No bootstrap required, opened a new log
I20260929 20:47:29.835973 8493 ts_tablet_manager.cc:1397] T bba0feb2d6f94aab92d201273c33f713 P 2fb8841de3304be998e952b840b3bc7a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 20:47:29.836767 8493 raft_consensus.cc:348] T bba0feb2d6f94aab92d201273c33f713 P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:29.836993 8493 raft_consensus.cc:374] T bba0feb2d6f94aab92d201273c33f713 P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:29.837109 8493 raft_consensus.cc:729] T bba0feb2d6f94aab92d201273c33f713 P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb8841de3304be998e952b840b3bc7a, State: Initialized, Role: FOLLOWER
I20260929 20:47:29.837409 8493 consensus_queue.cc:260] T bba0feb2d6f94aab92d201273c33f713 P 2fb8841de3304be998e952b840b3bc7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:29.838464 8494 ts_tablet_manager.cc:1428] T bba0feb2d6f94aab92d201273c33f713 P f37db28cb1064e75ac5f7ec8c6d6f677: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260929 20:47:29.838666 8493 ts_tablet_manager.cc:1428] T bba0feb2d6f94aab92d201273c33f713 P 2fb8841de3304be998e952b840b3bc7a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 20:47:29.972332 8500 raft_consensus.cc:482] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:47:29.972587 8500 raft_consensus.cc:504] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:29.974419 8500 leader_election.cc:288] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f37db28cb1064e75ac5f7ec8c6d6f677 (127.0.97.2:45097), 2fb8841de3304be998e952b840b3bc7a (127.0.97.3:44777)
I20260929 20:47:29.983273 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:29.975305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38859 (local address 127.0.97.3:44777)
0929 20:47:29.975461 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:29.975467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:29.975980 (+ 513us) server_negotiation.cc:407] Connection header received
0929 20:47:29.976276 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:29.976283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:29.976367 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:29.976449 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:29.976979 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.978060 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.980634 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.981886 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.981976 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:29.982076 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:29.982888 (+ 812us) server_negotiation.cc:299] Negotiation successful
0929 20:47:29.983042 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:47:29.984215 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:29.974637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44777 (local address 127.0.97.1:38859)
0929 20:47:29.975874 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:29.975896 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:29.976092 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:29.976481 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:29.976486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:29.976512 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:29.976811 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:29.976822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.978249 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.978255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:29.980453 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:29.980467 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.982545 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.982551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:29.982720 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:29.982750 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:47:29.983746 (+ 996us) client_negotiation.cc:240] Negotiation successful
0929 20:47:29.983893 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"thread_start_us":955,"threads_started":1}
I20260929 20:47:29.984498 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:29.974938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55463 (local address 127.0.97.2:45097)
0929 20:47:29.975139 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:29.975146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:29.976379 (+ 1233us) server_negotiation.cc:407] Connection header received
0929 20:47:29.976565 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:29.976570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:29.976649 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:29.976748 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:29.978565 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.979766 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.982017 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.983336 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.983442 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:29.983527 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:29.984080 (+ 553us) server_negotiation.cc:299] Negotiation successful
0929 20:47:29.984209 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:47:29.985087 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:29.974673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45097 (local address 127.0.97.1:55463)
0929 20:47:29.976264 (+ 1591us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:29.976280 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:29.976407 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:29.977942 (+ 1535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:29.977947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:29.977975 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:29.978363 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:29.978375 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.979947 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.979952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:29.981857 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:29.981872 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:29.983729 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:29.983735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:29.983920 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:29.983951 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:47:29.984666 (+ 715us) client_negotiation.cc:240] Negotiation successful
0929 20:47:29.984801 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1478,"thread_start_us":1260,"threads_started":1}
I20260929 20:47:29.985188 8419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba0feb2d6f94aab92d201273c33f713" candidate_uuid: "5086db73455d42d6ba8ecdae4169f287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fb8841de3304be998e952b840b3bc7a" is_pre_election: true
I20260929 20:47:29.985440 8419 raft_consensus.cc:2454] T bba0feb2d6f94aab92d201273c33f713 P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5086db73455d42d6ba8ecdae4169f287 in term 0.
I20260929 20:47:29.985972 8074 leader_election.cc:302] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb8841de3304be998e952b840b3bc7a, 5086db73455d42d6ba8ecdae4169f287; no voters:
I20260929 20:47:29.985973 8285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba0feb2d6f94aab92d201273c33f713" candidate_uuid: "5086db73455d42d6ba8ecdae4169f287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" is_pre_election: true
I20260929 20:47:29.986384 8500 raft_consensus.cc:2781] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:47:29.986382 8285 raft_consensus.cc:2454] T bba0feb2d6f94aab92d201273c33f713 P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5086db73455d42d6ba8ecdae4169f287 in term 0.
I20260929 20:47:29.986496 8500 raft_consensus.cc:482] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:47:29.986577 8500 raft_consensus.cc:3037] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:29.989607 8500 raft_consensus.cc:504] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:29.990635 8500 leader_election.cc:288] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287 [CANDIDATE]: Term 1 election: Requested vote from peers f37db28cb1064e75ac5f7ec8c6d6f677 (127.0.97.2:45097), 2fb8841de3304be998e952b840b3bc7a (127.0.97.3:44777)
I20260929 20:47:29.990648 8285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba0feb2d6f94aab92d201273c33f713" candidate_uuid: "5086db73455d42d6ba8ecdae4169f287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677"
I20260929 20:47:29.990839 8285 raft_consensus.cc:3037] T bba0feb2d6f94aab92d201273c33f713 P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:29.990981 8419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba0feb2d6f94aab92d201273c33f713" candidate_uuid: "5086db73455d42d6ba8ecdae4169f287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fb8841de3304be998e952b840b3bc7a"
I20260929 20:47:29.991191 8419 raft_consensus.cc:3037] T bba0feb2d6f94aab92d201273c33f713 P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:29.993896 8419 raft_consensus.cc:2454] T bba0feb2d6f94aab92d201273c33f713 P 2fb8841de3304be998e952b840b3bc7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5086db73455d42d6ba8ecdae4169f287 in term 1.
I20260929 20:47:29.994078 8285 raft_consensus.cc:2454] T bba0feb2d6f94aab92d201273c33f713 P f37db28cb1064e75ac5f7ec8c6d6f677 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5086db73455d42d6ba8ecdae4169f287 in term 1.
I20260929 20:47:29.994329 8074 leader_election.cc:302] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb8841de3304be998e952b840b3bc7a, 5086db73455d42d6ba8ecdae4169f287; no voters:
I20260929 20:47:29.994732 8500 raft_consensus.cc:2781] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:47:29.995200 8500 raft_consensus.cc:686] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287 [term 1 LEADER]: Becoming Leader. State: Replica: 5086db73455d42d6ba8ecdae4169f287, State: Running, Role: LEADER
I20260929 20:47:29.995595 8500 consensus_queue.cc:237] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:29.999214 7983 catalog_manager.cc:5495] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5086db73455d42d6ba8ecdae4169f287 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "5086db73455d42d6ba8ecdae4169f287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } health_report { overall_health: UNKNOWN } } }
Sep 29 20:47:30 dist-test-slave-blv7 krb5kdc[7939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714847, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:47:30.069288 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:30.053338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35305 (local address 127.0.0.1:55852)
0929 20:47:30.054067 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:30.054088 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:30.054255 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:30.054660 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:30.054665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:30.055714 (+ 1049us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:30.056054 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:30.056067 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.057148 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.057154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:30.058658 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:30.058673 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.059423 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.059434 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:30.059601 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:30.060323 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:30.060347 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:30.063283 (+ 2936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:30.066408 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:30.066423 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:30.066432 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:30.067051 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:30.067420 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:30.067425 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:30.067429 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:30.067590 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:30.068130 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:30.068139 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:30.068352 (+ 213us) client_negotiation.cc:769] Sending connection context
0929 20:47:30.068528 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 20:47:30.068864 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":378,"threads_started":1}
I20260929 20:47:30.069329 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:30.053338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55852 (local address 127.0.97.62:35305)
0929 20:47:30.054368 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:30.054372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:30.054391 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:47:30.054435 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:30.054438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:30.054489 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:30.054574 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:30.056241 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.057015 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.058804 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.059238 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.059335 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:30.063499 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:30.063504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:30.063507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:30.063563 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:30.066275 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:30.067162 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:30.067166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:30.067168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:30.067318 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:30.067725 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:30.067729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:30.067732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:30.067999 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:30.068105 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:30.068631 (+ 526us) server_negotiation.cc:299] Negotiation successful
0929 20:47:30.069034 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":189,"server-negotiator.queue_time_us":935,"thread_start_us":769,"threads_started":1}
I20260929 20:47:30.074635 7983 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55852:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 20:47:30.075660 7983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:47:30.082391 8121 tablet_service.cc:1467] Processing CreateTablet for tablet 6f57b5e601514f9cb15d96f438eda0fd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=003c0489d2374d0da28e794a9d0d71fc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:47:30.082396 8260 tablet_service.cc:1467] Processing CreateTablet for tablet 6f57b5e601514f9cb15d96f438eda0fd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=003c0489d2374d0da28e794a9d0d71fc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:47:30.082994 8121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f57b5e601514f9cb15d96f438eda0fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:30.083007 8260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f57b5e601514f9cb15d96f438eda0fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:30.087765 8399 tablet_service.cc:1467] Processing CreateTablet for tablet 6f57b5e601514f9cb15d96f438eda0fd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=003c0489d2374d0da28e794a9d0d71fc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:47:30.088351 8399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f57b5e601514f9cb15d96f438eda0fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:30.089102 8494 tablet_bootstrap.cc:492] T 6f57b5e601514f9cb15d96f438eda0fd P f37db28cb1064e75ac5f7ec8c6d6f677: Bootstrap starting.
I20260929 20:47:30.090427 8494 tablet_bootstrap.cc:654] T 6f57b5e601514f9cb15d96f438eda0fd P f37db28cb1064e75ac5f7ec8c6d6f677: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:30.092604 8494 tablet_bootstrap.cc:492] T 6f57b5e601514f9cb15d96f438eda0fd P f37db28cb1064e75ac5f7ec8c6d6f677: No bootstrap required, opened a new log
I20260929 20:47:30.092753 8494 ts_tablet_manager.cc:1397] T 6f57b5e601514f9cb15d96f438eda0fd P f37db28cb1064e75ac5f7ec8c6d6f677: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 20:47:30.093338 8494 raft_consensus.cc:348] T 6f57b5e601514f9cb15d96f438eda0fd P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.093528 8494 raft_consensus.cc:374] T 6f57b5e601514f9cb15d96f438eda0fd P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:30.093608 8494 raft_consensus.cc:729] T 6f57b5e601514f9cb15d96f438eda0fd P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f37db28cb1064e75ac5f7ec8c6d6f677, State: Initialized, Role: FOLLOWER
I20260929 20:47:30.093866 8494 consensus_queue.cc:260] T 6f57b5e601514f9cb15d96f438eda0fd P f37db28cb1064e75ac5f7ec8c6d6f677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.094414 8494 ts_tablet_manager.cc:1428] T 6f57b5e601514f9cb15d96f438eda0fd P f37db28cb1064e75ac5f7ec8c6d6f677: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 20:47:30.094245 8504 tablet_replica.cc:439] T 6f57b5e601514f9cb15d96f438eda0fd P f37db28cb1064e75ac5f7ec8c6d6f677: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } } }
I20260929 20:47:30.094563 8504 tablet_replica.cc:442] T 6f57b5e601514f9cb15d96f438eda0fd P f37db28cb1064e75ac5f7ec8c6d6f677: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:47:30.096552 8493 tablet_bootstrap.cc:492] T 6f57b5e601514f9cb15d96f438eda0fd P 2fb8841de3304be998e952b840b3bc7a: Bootstrap starting.
I20260929 20:47:30.097875 8493 tablet_bootstrap.cc:654] T 6f57b5e601514f9cb15d96f438eda0fd P 2fb8841de3304be998e952b840b3bc7a: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:30.098057 8495 tablet_bootstrap.cc:492] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287: Bootstrap starting.
I20260929 20:47:30.099738 8495 tablet_bootstrap.cc:654] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:30.101332 8493 tablet_bootstrap.cc:492] T 6f57b5e601514f9cb15d96f438eda0fd P 2fb8841de3304be998e952b840b3bc7a: No bootstrap required, opened a new log
I20260929 20:47:30.101483 8493 ts_tablet_manager.cc:1397] T 6f57b5e601514f9cb15d96f438eda0fd P 2fb8841de3304be998e952b840b3bc7a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 20:47:30.102177 8493 raft_consensus.cc:348] T 6f57b5e601514f9cb15d96f438eda0fd P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.102439 8493 raft_consensus.cc:374] T 6f57b5e601514f9cb15d96f438eda0fd P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:30.102515 8493 raft_consensus.cc:729] T 6f57b5e601514f9cb15d96f438eda0fd P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb8841de3304be998e952b840b3bc7a, State: Initialized, Role: FOLLOWER
I20260929 20:47:30.102753 8493 consensus_queue.cc:260] T 6f57b5e601514f9cb15d96f438eda0fd P 2fb8841de3304be998e952b840b3bc7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.103216 8499 tablet_replica.cc:439] T 6f57b5e601514f9cb15d96f438eda0fd P 2fb8841de3304be998e952b840b3bc7a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } } }
I20260929 20:47:30.103523 8499 tablet_replica.cc:442] T 6f57b5e601514f9cb15d96f438eda0fd P 2fb8841de3304be998e952b840b3bc7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:47:30.104452 8493 ts_tablet_manager.cc:1428] T 6f57b5e601514f9cb15d96f438eda0fd P 2fb8841de3304be998e952b840b3bc7a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 20:47:30.110580 8495 tablet_bootstrap.cc:492] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287: No bootstrap required, opened a new log
I20260929 20:47:30.110749 8495 ts_tablet_manager.cc:1397] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260929 20:47:30.111444 8495 raft_consensus.cc:348] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.111644 8495 raft_consensus.cc:374] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:30.111711 8495 raft_consensus.cc:729] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5086db73455d42d6ba8ecdae4169f287, State: Initialized, Role: FOLLOWER
I20260929 20:47:30.111935 8495 consensus_queue.cc:260] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.112488 8495 ts_tablet_manager.cc:1428] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 20:47:30.112354 8500 tablet_replica.cc:439] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } } }
I20260929 20:47:30.112684 8500 tablet_replica.cc:442] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:47:30.140219 8500 raft_consensus.cc:482] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:47:30.140450 8500 raft_consensus.cc:504] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.141240 8500 leader_election.cc:288] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f37db28cb1064e75ac5f7ec8c6d6f677 (127.0.97.2:45097), 2fb8841de3304be998e952b840b3bc7a (127.0.97.3:44777)
I20260929 20:47:30.141772 8285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f57b5e601514f9cb15d96f438eda0fd" candidate_uuid: "5086db73455d42d6ba8ecdae4169f287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" is_pre_election: true
I20260929 20:47:30.141798 8419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f57b5e601514f9cb15d96f438eda0fd" candidate_uuid: "5086db73455d42d6ba8ecdae4169f287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fb8841de3304be998e952b840b3bc7a" is_pre_election: true
I20260929 20:47:30.142026 8285 raft_consensus.cc:2454] T 6f57b5e601514f9cb15d96f438eda0fd P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5086db73455d42d6ba8ecdae4169f287 in term 0.
I20260929 20:47:30.142048 8419 raft_consensus.cc:2454] T 6f57b5e601514f9cb15d96f438eda0fd P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5086db73455d42d6ba8ecdae4169f287 in term 0.
I20260929 20:47:30.142436 8073 leader_election.cc:302] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5086db73455d42d6ba8ecdae4169f287, f37db28cb1064e75ac5f7ec8c6d6f677; no voters:
I20260929 20:47:30.142766 8500 raft_consensus.cc:2781] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:47:30.142863 8500 raft_consensus.cc:482] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:47:30.142963 8500 raft_consensus.cc:3037] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:30.144932 8500 raft_consensus.cc:504] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.145915 8419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f57b5e601514f9cb15d96f438eda0fd" candidate_uuid: "5086db73455d42d6ba8ecdae4169f287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fb8841de3304be998e952b840b3bc7a"
I20260929 20:47:30.145939 8285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f57b5e601514f9cb15d96f438eda0fd" candidate_uuid: "5086db73455d42d6ba8ecdae4169f287" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677"
I20260929 20:47:30.146135 8419 raft_consensus.cc:3037] T 6f57b5e601514f9cb15d96f438eda0fd P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:30.146162 8285 raft_consensus.cc:3037] T 6f57b5e601514f9cb15d96f438eda0fd P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:30.147130 8500 leader_election.cc:288] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287 [CANDIDATE]: Term 1 election: Requested vote from peers f37db28cb1064e75ac5f7ec8c6d6f677 (127.0.97.2:45097), 2fb8841de3304be998e952b840b3bc7a (127.0.97.3:44777)
I20260929 20:47:30.148486 8285 raft_consensus.cc:2454] T 6f57b5e601514f9cb15d96f438eda0fd P f37db28cb1064e75ac5f7ec8c6d6f677 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5086db73455d42d6ba8ecdae4169f287 in term 1.
I20260929 20:47:30.148834 8073 leader_election.cc:302] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5086db73455d42d6ba8ecdae4169f287, f37db28cb1064e75ac5f7ec8c6d6f677; no voters:
I20260929 20:47:30.148968 8419 raft_consensus.cc:2454] T 6f57b5e601514f9cb15d96f438eda0fd P 2fb8841de3304be998e952b840b3bc7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5086db73455d42d6ba8ecdae4169f287 in term 1.
I20260929 20:47:30.149092 8500 raft_consensus.cc:2781] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:47:30.149533 8500 raft_consensus.cc:686] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287 [term 1 LEADER]: Becoming Leader. State: Replica: 5086db73455d42d6ba8ecdae4169f287, State: Running, Role: LEADER
I20260929 20:47:30.149776 8500 consensus_queue.cc:237] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.151466 8524 tablet_replica.cc:439] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287: TxnStatusTablet state changed. Reason: New leader 5086db73455d42d6ba8ecdae4169f287. Latest consensus state: current_term: 1 leader_uuid: "5086db73455d42d6ba8ecdae4169f287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } } }
I20260929 20:47:30.151888 8524 tablet_replica.cc:442] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:47:30.152518 8537 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 20:47:30.152719 8537 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 20:47:30.153066 7983 catalog_manager.cc:5495] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5086db73455d42d6ba8ecdae4169f287 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "5086db73455d42d6ba8ecdae4169f287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } health_report { overall_health: UNKNOWN } } }
I20260929 20:47:30.173574 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:30.168167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48294 (local address 127.0.97.1:38499)
0929 20:47:30.168344 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:30.168350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:30.168452 (+ 102us) server_negotiation.cc:407] Connection header received
0929 20:47:30.168712 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:30.168716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:30.168817 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:30.168938 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:47:30.169321 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.170087 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.171417 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.171872 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.171981 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:30.172404 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:47:30.172922 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:47:30.173050 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:30.173153 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 20:47:30.173229 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:47:30.173566 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:30.168054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38499 (local address 127.0.0.1:48294)
0929 20:47:30.168369 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:30.168387 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:30.168525 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:30.168991 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:30.168995 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:30.169007 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:47:30.169186 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:30.169194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.170214 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.170220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:30.171288 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:30.171302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.172011 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.172017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:30.172177 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:30.172229 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:47:30.173024 (+ 795us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:47:30.173044 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 20:47:30.173123 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 20:47:30.173266 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260929 20:47:30.177238 8419 raft_consensus.cc:1261] T 6f57b5e601514f9cb15d96f438eda0fd P 2fb8841de3304be998e952b840b3bc7a [term 1 FOLLOWER]: Refusing update from remote peer 5086db73455d42d6ba8ecdae4169f287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:47:30.177529 8285 raft_consensus.cc:1261] T 6f57b5e601514f9cb15d96f438eda0fd P f37db28cb1064e75ac5f7ec8c6d6f677 [term 1 FOLLOWER]: Refusing update from remote peer 5086db73455d42d6ba8ecdae4169f287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:47:30.178009 8500 consensus_queue.cc:1035] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:47:30.178810 8500 consensus_queue.cc:1035] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:47:30.191960 8499 tablet_replica.cc:439] T 6f57b5e601514f9cb15d96f438eda0fd P 2fb8841de3304be998e952b840b3bc7a: TxnStatusTablet state changed. Reason: New leader 5086db73455d42d6ba8ecdae4169f287. Latest consensus state: current_term: 1 leader_uuid: "5086db73455d42d6ba8ecdae4169f287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } } }
I20260929 20:47:30.192265 8499 tablet_replica.cc:442] T 6f57b5e601514f9cb15d96f438eda0fd P 2fb8841de3304be998e952b840b3bc7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:47:30.196344 8500 tablet_replica.cc:439] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5086db73455d42d6ba8ecdae4169f287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } } }
I20260929 20:47:30.196667 8500 tablet_replica.cc:442] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:47:30.197366 8504 tablet_replica.cc:439] T 6f57b5e601514f9cb15d96f438eda0fd P f37db28cb1064e75ac5f7ec8c6d6f677: TxnStatusTablet state changed. Reason: New leader 5086db73455d42d6ba8ecdae4169f287. Latest consensus state: current_term: 1 leader_uuid: "5086db73455d42d6ba8ecdae4169f287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } } }
I20260929 20:47:30.197638 8504 tablet_replica.cc:442] T 6f57b5e601514f9cb15d96f438eda0fd P f37db28cb1064e75ac5f7ec8c6d6f677: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:47:30.200204 8499 tablet_replica.cc:439] T 6f57b5e601514f9cb15d96f438eda0fd P 2fb8841de3304be998e952b840b3bc7a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5086db73455d42d6ba8ecdae4169f287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } } }
I20260929 20:47:30.200421 8499 tablet_replica.cc:442] T 6f57b5e601514f9cb15d96f438eda0fd P 2fb8841de3304be998e952b840b3bc7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:47:30.204798 8524 tablet_replica.cc:439] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5086db73455d42d6ba8ecdae4169f287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } } }
I20260929 20:47:30.205065 8524 tablet_replica.cc:442] T 6f57b5e601514f9cb15d96f438eda0fd P 5086db73455d42d6ba8ecdae4169f287: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:47:30.209156 8504 tablet_replica.cc:439] T 6f57b5e601514f9cb15d96f438eda0fd P f37db28cb1064e75ac5f7ec8c6d6f677: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5086db73455d42d6ba8ecdae4169f287" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } } }
I20260929 20:47:30.209432 8504 tablet_replica.cc:442] T 6f57b5e601514f9cb15d96f438eda0fd P f37db28cb1064e75ac5f7ec8c6d6f677: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:47:30.209692 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:30.203048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48298 (local address 127.0.97.1:38499)
0929 20:47:30.203206 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:30.203211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:30.203610 (+ 399us) server_negotiation.cc:407] Connection header received
0929 20:47:30.203696 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:30.203701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:30.203773 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:30.203846 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:47:30.204353 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.205473 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.207321 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.207967 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.208076 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:30.208638 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:47:30.208754 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:47:30.208827 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:30.209345 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 20:47:30.209415 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:47:30.233256 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:30.225897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38499 (local address 127.0.97.1:42755)
0929 20:47:30.226613 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:30.226632 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:30.226824 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:30.227211 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:30.227216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:30.227229 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:47:30.227594 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:30.227605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.229031 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.229036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:30.230947 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:30.230961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.231982 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.231988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:30.232155 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:30.232221 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:47:30.232637 (+ 416us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:47:30.232644 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:47:30.232709 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 20:47:30.232838 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":579,"threads_started":1}
I20260929 20:47:30.233853 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:30.226272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42755 (local address 127.0.97.1:38499)
0929 20:47:30.226479 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:30.226485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:30.226707 (+ 222us) server_negotiation.cc:407] Connection header received
0929 20:47:30.226978 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:30.226983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:30.227089 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:30.227189 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:47:30.227774 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.228879 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.231090 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.231795 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.231909 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:30.232385 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:47:30.232530 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:47:30.232605 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:30.233527 (+ 922us) server_negotiation.cc:299] Negotiation successful
0929 20:47:30.233595 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":75}
I20260929 20:47:30.245023 8419 raft_consensus.cc:1261] T bba0feb2d6f94aab92d201273c33f713 P 2fb8841de3304be998e952b840b3bc7a [term 1 FOLLOWER]: Refusing update from remote peer 5086db73455d42d6ba8ecdae4169f287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:47:30.245048 8285 raft_consensus.cc:1261] T bba0feb2d6f94aab92d201273c33f713 P f37db28cb1064e75ac5f7ec8c6d6f677 [term 1 FOLLOWER]: Refusing update from remote peer 5086db73455d42d6ba8ecdae4169f287: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:47:30.245558 8542 consensus_queue.cc:1035] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:47:30.245931 8500 consensus_queue.cc:1035] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:47:30.306023 7983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:47:30.306236 7983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:47:30.308928 7983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10: Sending DeleteTablet for 3 replicas of tablet bba0feb2d6f94aab92d201273c33f713
I20260929 20:47:30.309675 8260 tablet_service.cc:1514] Processing DeleteTablet for tablet bba0feb2d6f94aab92d201273c33f713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52660
I20260929 20:47:30.309746 8121 tablet_service.cc:1514] Processing DeleteTablet for tablet bba0feb2d6f94aab92d201273c33f713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48286
I20260929 20:47:30.309970 8517 tablet_replica.cc:330] T bba0feb2d6f94aab92d201273c33f713 P f37db28cb1064e75ac5f7ec8c6d6f677: stopping tablet replica
I20260929 20:47:30.309954 8399 tablet_service.cc:1514] Processing DeleteTablet for tablet bba0feb2d6f94aab92d201273c33f713 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48414
I20260929 20:47:30.310185 8518 tablet_replica.cc:330] T bba0feb2d6f94aab92d201273c33f713 P 2fb8841de3304be998e952b840b3bc7a: stopping tablet replica
I20260929 20:47:30.310158 8517 raft_consensus.cc:2229] T bba0feb2d6f94aab92d201273c33f713 P f37db28cb1064e75ac5f7ec8c6d6f677 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:47:30.310328 8517 raft_consensus.cc:2258] T bba0feb2d6f94aab92d201273c33f713 P f37db28cb1064e75ac5f7ec8c6d6f677 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:30.310340 8518 raft_consensus.cc:2229] T bba0feb2d6f94aab92d201273c33f713 P 2fb8841de3304be998e952b840b3bc7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:47:30.310494 8518 raft_consensus.cc:2258] T bba0feb2d6f94aab92d201273c33f713 P 2fb8841de3304be998e952b840b3bc7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:30.310572 8516 tablet_replica.cc:330] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287: stopping tablet replica
I20260929 20:47:30.310752 8516 raft_consensus.cc:2229] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:47:30.311555 8516 raft_consensus.cc:2258] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:30.311637 8518 ts_tablet_manager.cc:1905] T bba0feb2d6f94aab92d201273c33f713 P 2fb8841de3304be998e952b840b3bc7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:30.312980 8517 ts_tablet_manager.cc:1905] T bba0feb2d6f94aab92d201273c33f713 P f37db28cb1064e75ac5f7ec8c6d6f677: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:30.315874 8516 ts_tablet_manager.cc:1905] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:30.319690 8517 ts_tablet_manager.cc:1918] T bba0feb2d6f94aab92d201273c33f713 P f37db28cb1064e75ac5f7ec8c6d6f677: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:47:30.319916 8517 log.cc:1196] T bba0feb2d6f94aab92d201273c33f713 P f37db28cb1064e75ac5f7ec8c6d6f677: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/bba0feb2d6f94aab92d201273c33f713
I20260929 20:47:30.320356 8517 ts_tablet_manager.cc:1939] T bba0feb2d6f94aab92d201273c33f713 P f37db28cb1064e75ac5f7ec8c6d6f677: Deleting consensus metadata
I20260929 20:47:30.321720 7970 catalog_manager.cc:4841] TS f37db28cb1064e75ac5f7ec8c6d6f677 (127.0.97.2:45097): tablet bba0feb2d6f94aab92d201273c33f713 (table test-table [id=5876ba51ddec411388f36d67f8b5839a]) successfully deleted
I20260929 20:47:30.323374 8518 ts_tablet_manager.cc:1918] T bba0feb2d6f94aab92d201273c33f713 P 2fb8841de3304be998e952b840b3bc7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:47:30.323519 8518 log.cc:1196] T bba0feb2d6f94aab92d201273c33f713 P 2fb8841de3304be998e952b840b3bc7a: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/bba0feb2d6f94aab92d201273c33f713
I20260929 20:47:30.323868 8518 ts_tablet_manager.cc:1939] T bba0feb2d6f94aab92d201273c33f713 P 2fb8841de3304be998e952b840b3bc7a: Deleting consensus metadata
I20260929 20:47:30.324841 7971 catalog_manager.cc:4841] TS 2fb8841de3304be998e952b840b3bc7a (127.0.97.3:44777): tablet bba0feb2d6f94aab92d201273c33f713 (table test-table [id=5876ba51ddec411388f36d67f8b5839a]) successfully deleted
I20260929 20:47:30.326687 8516 ts_tablet_manager.cc:1918] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:47:30.326846 8516 log.cc:1196] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/bba0feb2d6f94aab92d201273c33f713
I20260929 20:47:30.327262 8516 ts_tablet_manager.cc:1939] T bba0feb2d6f94aab92d201273c33f713 P 5086db73455d42d6ba8ecdae4169f287: Deleting consensus metadata
I20260929 20:47:30.328173 7969 catalog_manager.cc:4841] TS 5086db73455d42d6ba8ecdae4169f287 (127.0.97.1:38499): tablet bba0feb2d6f94aab92d201273c33f713 (table test-table [id=5876ba51ddec411388f36d67f8b5839a]) successfully deleted
Sep 29 20:47:30 dist-test-slave-blv7 krb5kdc[7939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:47:30 dist-test-slave-blv7 krb5kdc[7939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:30 dist-test-slave-blv7 krb5kdc[7939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 20:47:30.371940 8531 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:30.361930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55866 (local address 127.0.97.62:35305)
0929 20:47:30.362136 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:30.362142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:30.362456 (+ 314us) server_negotiation.cc:407] Connection header received
0929 20:47:30.362603 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:30.362608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:30.362697 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:30.362807 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:30.364033 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.365165 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.366500 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.367199 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.367325 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:30.371661 (+ 4336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55866: BlockingRecv error: recv got EOF from 127.0.0.1:55866 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 20:47:30 dist-test-slave-blv7 krb5kdc[7939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 20:47:30.389242 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:30.374167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55872 (local address 127.0.97.62:35305)
0929 20:47:30.374332 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:30.374337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:30.374615 (+ 278us) server_negotiation.cc:407] Connection header received
0929 20:47:30.374778 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:30.374782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:30.374837 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:30.374911 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:30.376345 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.377043 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.378764 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.379192 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.379289 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:30.383479 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:30.383485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:30.383490 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:30.383529 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:30.386655 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:30.387445 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:30.387451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:30.387454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:30.387579 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:30.387950 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:30.387954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:30.387956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:30.388229 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:30.388466 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:30.388745 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 20:47:30.388949 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:47:30.392644 7983 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:47:30.393301 7983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:47:30.399870 8121 tablet_service.cc:1467] Processing CreateTablet for tablet cb8ec81a092f454c807727095c4c12eb (DEFAULT_TABLE table=test-table [id=075eccd2c8554e268f83173aefa46f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:30.400043 8260 tablet_service.cc:1467] Processing CreateTablet for tablet cb8ec81a092f454c807727095c4c12eb (DEFAULT_TABLE table=test-table [id=075eccd2c8554e268f83173aefa46f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:30.400306 8121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb8ec81a092f454c807727095c4c12eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:30.400214 8399 tablet_service.cc:1467] Processing CreateTablet for tablet cb8ec81a092f454c807727095c4c12eb (DEFAULT_TABLE table=test-table [id=075eccd2c8554e268f83173aefa46f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:30.400462 8260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb8ec81a092f454c807727095c4c12eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:30.400588 8399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb8ec81a092f454c807727095c4c12eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:30.406852 8493 tablet_bootstrap.cc:492] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a: Bootstrap starting.
I20260929 20:47:30.408445 8493 tablet_bootstrap.cc:654] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:30.409827 8494 tablet_bootstrap.cc:492] T cb8ec81a092f454c807727095c4c12eb P f37db28cb1064e75ac5f7ec8c6d6f677: Bootstrap starting.
I20260929 20:47:30.411324 8494 tablet_bootstrap.cc:654] T cb8ec81a092f454c807727095c4c12eb P f37db28cb1064e75ac5f7ec8c6d6f677: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:30.411446 8495 tablet_bootstrap.cc:492] T cb8ec81a092f454c807727095c4c12eb P 5086db73455d42d6ba8ecdae4169f287: Bootstrap starting.
I20260929 20:47:30.412482 8493 tablet_bootstrap.cc:492] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a: No bootstrap required, opened a new log
I20260929 20:47:30.412670 8493 ts_tablet_manager.cc:1397] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 20:47:30.413030 8495 tablet_bootstrap.cc:654] T cb8ec81a092f454c807727095c4c12eb P 5086db73455d42d6ba8ecdae4169f287: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:30.413553 8493 raft_consensus.cc:348] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.414048 8493 raft_consensus.cc:374] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:30.414218 8493 raft_consensus.cc:729] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb8841de3304be998e952b840b3bc7a, State: Initialized, Role: FOLLOWER
I20260929 20:47:30.414539 8493 consensus_queue.cc:260] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.417636 8493 ts_tablet_manager.cc:1428] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260929 20:47:30.422178 8495 tablet_bootstrap.cc:492] T cb8ec81a092f454c807727095c4c12eb P 5086db73455d42d6ba8ecdae4169f287: No bootstrap required, opened a new log
I20260929 20:47:30.422324 8495 ts_tablet_manager.cc:1397] T cb8ec81a092f454c807727095c4c12eb P 5086db73455d42d6ba8ecdae4169f287: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260929 20:47:30.423098 8495 raft_consensus.cc:348] T cb8ec81a092f454c807727095c4c12eb P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.423319 8495 raft_consensus.cc:374] T cb8ec81a092f454c807727095c4c12eb P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:30.423417 8495 raft_consensus.cc:729] T cb8ec81a092f454c807727095c4c12eb P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5086db73455d42d6ba8ecdae4169f287, State: Initialized, Role: FOLLOWER
I20260929 20:47:30.423552 8494 tablet_bootstrap.cc:492] T cb8ec81a092f454c807727095c4c12eb P f37db28cb1064e75ac5f7ec8c6d6f677: No bootstrap required, opened a new log
I20260929 20:47:30.423695 8494 ts_tablet_manager.cc:1397] T cb8ec81a092f454c807727095c4c12eb P f37db28cb1064e75ac5f7ec8c6d6f677: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260929 20:47:30.423703 8495 consensus_queue.cc:260] T cb8ec81a092f454c807727095c4c12eb P 5086db73455d42d6ba8ecdae4169f287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.424420 8494 raft_consensus.cc:348] T cb8ec81a092f454c807727095c4c12eb P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.424665 8494 raft_consensus.cc:374] T cb8ec81a092f454c807727095c4c12eb P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:30.424760 8494 raft_consensus.cc:729] T cb8ec81a092f454c807727095c4c12eb P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f37db28cb1064e75ac5f7ec8c6d6f677, State: Initialized, Role: FOLLOWER
I20260929 20:47:30.424924 8495 ts_tablet_manager.cc:1428] T cb8ec81a092f454c807727095c4c12eb P 5086db73455d42d6ba8ecdae4169f287: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:47:30.425010 8494 consensus_queue.cc:260] T cb8ec81a092f454c807727095c4c12eb P f37db28cb1064e75ac5f7ec8c6d6f677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.425550 8494 ts_tablet_manager.cc:1428] T cb8ec81a092f454c807727095c4c12eb P f37db28cb1064e75ac5f7ec8c6d6f677: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 20:47:30.507447 8499 raft_consensus.cc:482] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:47:30.507700 8499 raft_consensus.cc:504] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.509647 8499 leader_election.cc:288] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5086db73455d42d6ba8ecdae4169f287 (127.0.97.1:38499), f37db28cb1064e75ac5f7ec8c6d6f677 (127.0.97.2:45097)
I20260929 20:47:30.527647 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:30.510203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52825 (local address 127.0.97.2:45097)
0929 20:47:30.521554 (+ 11351us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:30.521562 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:30.522221 (+ 659us) server_negotiation.cc:407] Connection header received
0929 20:47:30.522556 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:30.522562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:30.522636 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:30.522722 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:30.523191 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.524070 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.525822 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.526698 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.526808 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:30.526905 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:30.527240 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 20:47:30.527363 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11157,"thread_start_us":11372,"threads_started":1}
I20260929 20:47:30.527721 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:30.509868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45097 (local address 127.0.97.3:52825)
0929 20:47:30.522079 (+ 12211us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:30.522108 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:30.522368 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:30.522752 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:30.522756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:30.522777 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:30.522989 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:30.523001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.524184 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.524189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:30.525655 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:30.525667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.526870 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.526876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:30.527094 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:30.527118 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:47:30.527207 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 20:47:30.527333 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12078,"mutex_wait_us":3,"thread_start_us":11963,"threads_started":1}
I20260929 20:47:30.528335 8284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8ec81a092f454c807727095c4c12eb" candidate_uuid: "2fb8841de3304be998e952b840b3bc7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" is_pre_election: true
I20260929 20:47:30.528533 8284 raft_consensus.cc:2454] T cb8ec81a092f454c807727095c4c12eb P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fb8841de3304be998e952b840b3bc7a in term 0.
I20260929 20:47:30.528975 8352 leader_election.cc:302] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb8841de3304be998e952b840b3bc7a, f37db28cb1064e75ac5f7ec8c6d6f677; no voters:
I20260929 20:47:30.529386 8499 raft_consensus.cc:2781] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:47:30.529500 8499 raft_consensus.cc:482] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:47:30.529595 8499 raft_consensus.cc:3037] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:30.530295 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:30.510244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34789 (local address 127.0.97.1:38499)
0929 20:47:30.510508 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:30.510515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:30.522596 (+ 12081us) server_negotiation.cc:407] Connection header received
0929 20:47:30.522760 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:30.522766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:30.522854 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:30.522967 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:47:30.523574 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.524732 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.527422 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.528772 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.528896 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:30.529008 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:30.529832 (+ 824us) server_negotiation.cc:299] Negotiation successful
0929 20:47:30.529978 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260929 20:47:30.530407 8565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:30.509897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38499 (local address 127.0.97.3:34789)
0929 20:47:30.522469 (+ 12572us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:30.522487 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:30.522612 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:30.523007 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:30.523011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:30.523057 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:47:30.523401 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:30.523412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.524916 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.524922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:30.527225 (+ 2303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:30.527241 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.529428 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.529435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:30.529642 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:30.529681 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 20:47:30.529787 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 20:47:30.529931 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12479,"mutex_wait_us":31,"thread_start_us":12293,"threads_started":1}
I20260929 20:47:30.531309 8141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8ec81a092f454c807727095c4c12eb" candidate_uuid: "2fb8841de3304be998e952b840b3bc7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5086db73455d42d6ba8ecdae4169f287" is_pre_election: true
I20260929 20:47:30.531613 8141 raft_consensus.cc:2454] T cb8ec81a092f454c807727095c4c12eb P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fb8841de3304be998e952b840b3bc7a in term 0.
I20260929 20:47:30.532441 8499 raft_consensus.cc:504] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.532974 8499 leader_election.cc:288] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a [CANDIDATE]: Term 1 election: Requested vote from peers 5086db73455d42d6ba8ecdae4169f287 (127.0.97.1:38499), f37db28cb1064e75ac5f7ec8c6d6f677 (127.0.97.2:45097)
I20260929 20:47:30.533416 8284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8ec81a092f454c807727095c4c12eb" candidate_uuid: "2fb8841de3304be998e952b840b3bc7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677"
I20260929 20:47:30.533579 8284 raft_consensus.cc:3037] T cb8ec81a092f454c807727095c4c12eb P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:30.535569 8284 raft_consensus.cc:2454] T cb8ec81a092f454c807727095c4c12eb P f37db28cb1064e75ac5f7ec8c6d6f677 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fb8841de3304be998e952b840b3bc7a in term 1.
I20260929 20:47:30.535913 8352 leader_election.cc:302] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb8841de3304be998e952b840b3bc7a, f37db28cb1064e75ac5f7ec8c6d6f677; no voters:
I20260929 20:47:30.536063 8141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb8ec81a092f454c807727095c4c12eb" candidate_uuid: "2fb8841de3304be998e952b840b3bc7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5086db73455d42d6ba8ecdae4169f287"
I20260929 20:47:30.536280 8499 raft_consensus.cc:2781] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:47:30.536301 8141 raft_consensus.cc:3037] T cb8ec81a092f454c807727095c4c12eb P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:30.536753 8499 raft_consensus.cc:686] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a [term 1 LEADER]: Becoming Leader. State: Replica: 2fb8841de3304be998e952b840b3bc7a, State: Running, Role: LEADER
I20260929 20:47:30.537288 8499 consensus_queue.cc:237] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.538760 8141 raft_consensus.cc:2454] T cb8ec81a092f454c807727095c4c12eb P 5086db73455d42d6ba8ecdae4169f287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fb8841de3304be998e952b840b3bc7a in term 1.
I20260929 20:47:30.549023 7983 catalog_manager.cc:5495] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fb8841de3304be998e952b840b3bc7a (127.0.97.3). New cstate: current_term: 1 leader_uuid: "2fb8841de3304be998e952b840b3bc7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } health_report { overall_health: HEALTHY } } }
I20260929 20:47:30.580639 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:30.575260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.0.97.3:44777)
0929 20:47:30.575822 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:30.575826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:30.575844 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:47:30.575925 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:30.575928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:30.575999 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:30.576140 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:47:30.576538 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.577285 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.578552 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.578970 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.579103 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:30.579502 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:47:30.580053 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:47:30.580178 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:30.580259 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 20:47:30.580331 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":293,"threads_started":1}
I20260929 20:47:30.584450 8284 raft_consensus.cc:1261] T cb8ec81a092f454c807727095c4c12eb P f37db28cb1064e75ac5f7ec8c6d6f677 [term 1 FOLLOWER]: Refusing update from remote peer 2fb8841de3304be998e952b840b3bc7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:47:30.584457 8141 raft_consensus.cc:1261] T cb8ec81a092f454c807727095c4c12eb P 5086db73455d42d6ba8ecdae4169f287 [term 1 FOLLOWER]: Refusing update from remote peer 2fb8841de3304be998e952b840b3bc7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:47:30.585170 8567 consensus_queue.cc:1035] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:47:30.585484 8499 consensus_queue.cc:1035] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:47:30.612789 7982 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:47:30.613039 7982 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:47:30.616427 7982 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10: Sending DeleteTablet for 3 replicas of tablet cb8ec81a092f454c807727095c4c12eb
I20260929 20:47:30.617488 8260 tablet_service.cc:1514] Processing DeleteTablet for tablet cb8ec81a092f454c807727095c4c12eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52660
I20260929 20:47:30.617491 8121 tablet_service.cc:1514] Processing DeleteTablet for tablet cb8ec81a092f454c807727095c4c12eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48286
I20260929 20:47:30.617762 8517 tablet_replica.cc:330] T cb8ec81a092f454c807727095c4c12eb P f37db28cb1064e75ac5f7ec8c6d6f677: stopping tablet replica
I20260929 20:47:30.617794 8516 tablet_replica.cc:330] T cb8ec81a092f454c807727095c4c12eb P 5086db73455d42d6ba8ecdae4169f287: stopping tablet replica
I20260929 20:47:30.617895 8517 raft_consensus.cc:2229] T cb8ec81a092f454c807727095c4c12eb P f37db28cb1064e75ac5f7ec8c6d6f677 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:47:30.617969 8516 raft_consensus.cc:2229] T cb8ec81a092f454c807727095c4c12eb P 5086db73455d42d6ba8ecdae4169f287 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:47:30.618064 8517 raft_consensus.cc:2258] T cb8ec81a092f454c807727095c4c12eb P f37db28cb1064e75ac5f7ec8c6d6f677 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:30.618201 8516 raft_consensus.cc:2258] T cb8ec81a092f454c807727095c4c12eb P 5086db73455d42d6ba8ecdae4169f287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:30.619256 8399 tablet_service.cc:1514] Processing DeleteTablet for tablet cb8ec81a092f454c807727095c4c12eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48414
I20260929 20:47:30.619572 8518 tablet_replica.cc:330] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a: stopping tablet replica
I20260929 20:47:30.619735 8518 raft_consensus.cc:2229] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:47:30.620229 8518 raft_consensus.cc:2258] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:30.620632 8516 ts_tablet_manager.cc:1905] T cb8ec81a092f454c807727095c4c12eb P 5086db73455d42d6ba8ecdae4169f287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:30.621232 8518 ts_tablet_manager.cc:1905] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:30.624791 7982 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:47:30.625528 7982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:47:30.630319 8518 ts_tablet_manager.cc:1918] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:47:30.630445 8518 log.cc:1196] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/cb8ec81a092f454c807727095c4c12eb
I20260929 20:47:30.630864 8518 ts_tablet_manager.cc:1939] T cb8ec81a092f454c807727095c4c12eb P 2fb8841de3304be998e952b840b3bc7a: Deleting consensus metadata
I20260929 20:47:30.631906 7971 catalog_manager.cc:4841] TS 2fb8841de3304be998e952b840b3bc7a (127.0.97.3:44777): tablet cb8ec81a092f454c807727095c4c12eb (table test-table [id=075eccd2c8554e268f83173aefa46f40]) successfully deleted
I20260929 20:47:30.633206 8260 tablet_service.cc:1467] Processing CreateTablet for tablet 3c07a7c0304240cfa9e204a9a711910e (DEFAULT_TABLE table=test-table [id=f423e425820744129dc2bbb7df97c2d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:30.633279 8121 tablet_service.cc:1467] Processing CreateTablet for tablet 3c07a7c0304240cfa9e204a9a711910e (DEFAULT_TABLE table=test-table [id=f423e425820744129dc2bbb7df97c2d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:30.633440 8399 tablet_service.cc:1467] Processing CreateTablet for tablet 3c07a7c0304240cfa9e204a9a711910e (DEFAULT_TABLE table=test-table [id=f423e425820744129dc2bbb7df97c2d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:47:30.633638 8121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c07a7c0304240cfa9e204a9a711910e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:30.633632 8260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c07a7c0304240cfa9e204a9a711910e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:30.633844 8399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c07a7c0304240cfa9e204a9a711910e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:30.639638 8494 tablet_bootstrap.cc:492] T 3c07a7c0304240cfa9e204a9a711910e P f37db28cb1064e75ac5f7ec8c6d6f677: Bootstrap starting.
I20260929 20:47:30.639973 8495 tablet_bootstrap.cc:492] T 3c07a7c0304240cfa9e204a9a711910e P 5086db73455d42d6ba8ecdae4169f287: Bootstrap starting.
I20260929 20:47:30.641178 8495 tablet_bootstrap.cc:654] T 3c07a7c0304240cfa9e204a9a711910e P 5086db73455d42d6ba8ecdae4169f287: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:30.641320 8494 tablet_bootstrap.cc:654] T 3c07a7c0304240cfa9e204a9a711910e P f37db28cb1064e75ac5f7ec8c6d6f677: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:30.643675 8493 tablet_bootstrap.cc:492] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a: Bootstrap starting.
I20260929 20:47:30.644145 8495 tablet_bootstrap.cc:492] T 3c07a7c0304240cfa9e204a9a711910e P 5086db73455d42d6ba8ecdae4169f287: No bootstrap required, opened a new log
I20260929 20:47:30.644304 8495 ts_tablet_manager.cc:1397] T 3c07a7c0304240cfa9e204a9a711910e P 5086db73455d42d6ba8ecdae4169f287: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 20:47:30.645262 8493 tablet_bootstrap.cc:654] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:30.645115 8495 raft_consensus.cc:348] T 3c07a7c0304240cfa9e204a9a711910e P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.645375 8495 raft_consensus.cc:374] T 3c07a7c0304240cfa9e204a9a711910e P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:30.645471 8495 raft_consensus.cc:729] T 3c07a7c0304240cfa9e204a9a711910e P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5086db73455d42d6ba8ecdae4169f287, State: Initialized, Role: FOLLOWER
I20260929 20:47:30.645771 8495 consensus_queue.cc:260] T 3c07a7c0304240cfa9e204a9a711910e P 5086db73455d42d6ba8ecdae4169f287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.646847 8516 ts_tablet_manager.cc:1918] T cb8ec81a092f454c807727095c4c12eb P 5086db73455d42d6ba8ecdae4169f287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:47:30.646852 8495 ts_tablet_manager.cc:1428] T 3c07a7c0304240cfa9e204a9a711910e P 5086db73455d42d6ba8ecdae4169f287: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260929 20:47:30.648002 8516 log.cc:1196] T cb8ec81a092f454c807727095c4c12eb P 5086db73455d42d6ba8ecdae4169f287: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/cb8ec81a092f454c807727095c4c12eb
I20260929 20:47:30.649818 8517 ts_tablet_manager.cc:1905] T cb8ec81a092f454c807727095c4c12eb P f37db28cb1064e75ac5f7ec8c6d6f677: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:30.656340 8517 ts_tablet_manager.cc:1918] T cb8ec81a092f454c807727095c4c12eb P f37db28cb1064e75ac5f7ec8c6d6f677: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:47:30.656541 8517 log.cc:1196] T cb8ec81a092f454c807727095c4c12eb P f37db28cb1064e75ac5f7ec8c6d6f677: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/cb8ec81a092f454c807727095c4c12eb
I20260929 20:47:30.656980 8517 ts_tablet_manager.cc:1939] T cb8ec81a092f454c807727095c4c12eb P f37db28cb1064e75ac5f7ec8c6d6f677: Deleting consensus metadata
I20260929 20:47:30.658093 7970 catalog_manager.cc:4841] TS f37db28cb1064e75ac5f7ec8c6d6f677 (127.0.97.2:45097): tablet cb8ec81a092f454c807727095c4c12eb (table test-table [id=075eccd2c8554e268f83173aefa46f40]) successfully deleted
I20260929 20:47:30.648666 8516 ts_tablet_manager.cc:1939] T cb8ec81a092f454c807727095c4c12eb P 5086db73455d42d6ba8ecdae4169f287: Deleting consensus metadata
I20260929 20:47:30.659839 7969 catalog_manager.cc:4841] TS 5086db73455d42d6ba8ecdae4169f287 (127.0.97.1:38499): tablet cb8ec81a092f454c807727095c4c12eb (table test-table [id=075eccd2c8554e268f83173aefa46f40]) successfully deleted
I20260929 20:47:30.661242 8493 tablet_bootstrap.cc:492] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a: No bootstrap required, opened a new log
I20260929 20:47:30.661428 8493 ts_tablet_manager.cc:1397] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.005s
I20260929 20:47:30.662247 8493 raft_consensus.cc:348] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.662521 8493 raft_consensus.cc:374] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:30.662618 8493 raft_consensus.cc:729] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb8841de3304be998e952b840b3bc7a, State: Initialized, Role: FOLLOWER
I20260929 20:47:30.662856 8493 consensus_queue.cc:260] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.663548 8494 tablet_bootstrap.cc:492] T 3c07a7c0304240cfa9e204a9a711910e P f37db28cb1064e75ac5f7ec8c6d6f677: No bootstrap required, opened a new log
I20260929 20:47:30.663760 8494 ts_tablet_manager.cc:1397] T 3c07a7c0304240cfa9e204a9a711910e P f37db28cb1064e75ac5f7ec8c6d6f677: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.003s
I20260929 20:47:30.664898 8494 raft_consensus.cc:348] T 3c07a7c0304240cfa9e204a9a711910e P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.665273 8494 raft_consensus.cc:374] T 3c07a7c0304240cfa9e204a9a711910e P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:30.665387 8494 raft_consensus.cc:729] T 3c07a7c0304240cfa9e204a9a711910e P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f37db28cb1064e75ac5f7ec8c6d6f677, State: Initialized, Role: FOLLOWER
I20260929 20:47:30.665702 8494 consensus_queue.cc:260] T 3c07a7c0304240cfa9e204a9a711910e P f37db28cb1064e75ac5f7ec8c6d6f677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.666637 8493 ts_tablet_manager.cc:1428] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260929 20:47:30.668619 8494 ts_tablet_manager.cc:1428] T 3c07a7c0304240cfa9e204a9a711910e P f37db28cb1064e75ac5f7ec8c6d6f677: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260929 20:47:30.756428 8574 raft_consensus.cc:482] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:47:30.756731 8574 raft_consensus.cc:504] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.757726 8574 leader_election.cc:288] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5086db73455d42d6ba8ecdae4169f287 (127.0.97.1:38499), f37db28cb1064e75ac5f7ec8c6d6f677 (127.0.97.2:45097)
I20260929 20:47:30.758284 8141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c07a7c0304240cfa9e204a9a711910e" candidate_uuid: "2fb8841de3304be998e952b840b3bc7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5086db73455d42d6ba8ecdae4169f287" is_pre_election: true
I20260929 20:47:30.758453 8284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c07a7c0304240cfa9e204a9a711910e" candidate_uuid: "2fb8841de3304be998e952b840b3bc7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" is_pre_election: true
I20260929 20:47:30.758528 8141 raft_consensus.cc:2454] T 3c07a7c0304240cfa9e204a9a711910e P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fb8841de3304be998e952b840b3bc7a in term 0.
I20260929 20:47:30.758695 8284 raft_consensus.cc:2454] T 3c07a7c0304240cfa9e204a9a711910e P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fb8841de3304be998e952b840b3bc7a in term 0.
I20260929 20:47:30.759193 8351 leader_election.cc:302] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb8841de3304be998e952b840b3bc7a, 5086db73455d42d6ba8ecdae4169f287; no voters:
I20260929 20:47:30.759536 8574 raft_consensus.cc:2781] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:47:30.759665 8574 raft_consensus.cc:482] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:47:30.759758 8574 raft_consensus.cc:3037] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:30.762995 8574 raft_consensus.cc:504] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.764247 8284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c07a7c0304240cfa9e204a9a711910e" candidate_uuid: "2fb8841de3304be998e952b840b3bc7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677"
I20260929 20:47:30.764258 8141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c07a7c0304240cfa9e204a9a711910e" candidate_uuid: "2fb8841de3304be998e952b840b3bc7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5086db73455d42d6ba8ecdae4169f287"
I20260929 20:47:30.764457 8284 raft_consensus.cc:3037] T 3c07a7c0304240cfa9e204a9a711910e P f37db28cb1064e75ac5f7ec8c6d6f677 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:30.764513 8141 raft_consensus.cc:3037] T 3c07a7c0304240cfa9e204a9a711910e P 5086db73455d42d6ba8ecdae4169f287 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:30.767344 8284 raft_consensus.cc:2454] T 3c07a7c0304240cfa9e204a9a711910e P f37db28cb1064e75ac5f7ec8c6d6f677 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fb8841de3304be998e952b840b3bc7a in term 1.
I20260929 20:47:30.767678 8141 raft_consensus.cc:2454] T 3c07a7c0304240cfa9e204a9a711910e P 5086db73455d42d6ba8ecdae4169f287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fb8841de3304be998e952b840b3bc7a in term 1.
I20260929 20:47:30.767855 8352 leader_election.cc:302] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb8841de3304be998e952b840b3bc7a, f37db28cb1064e75ac5f7ec8c6d6f677; no voters:
I20260929 20:47:30.768152 8567 raft_consensus.cc:2781] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:47:30.768321 8567 raft_consensus.cc:686] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a [term 1 LEADER]: Becoming Leader. State: Replica: 2fb8841de3304be998e952b840b3bc7a, State: Running, Role: LEADER
I20260929 20:47:30.768606 8567 consensus_queue.cc:237] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } }
I20260929 20:47:30.768615 8574 leader_election.cc:288] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a [CANDIDATE]: Term 1 election: Requested vote from peers 5086db73455d42d6ba8ecdae4169f287 (127.0.97.1:38499), f37db28cb1064e75ac5f7ec8c6d6f677 (127.0.97.2:45097)
I20260929 20:47:30.774022 7983 catalog_manager.cc:5495] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fb8841de3304be998e952b840b3bc7a (127.0.97.3). New cstate: current_term: 1 leader_uuid: "2fb8841de3304be998e952b840b3bc7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fb8841de3304be998e952b840b3bc7a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44777 } health_report { overall_health: HEALTHY } } }
I20260929 20:47:30.847978 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:30.840487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36683 (local address 127.0.97.1:38499)
0929 20:47:30.840667 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:30.840673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:30.841385 (+ 712us) server_negotiation.cc:407] Connection header received
0929 20:47:30.841673 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:30.841683 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:30.841772 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:30.841868 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:47:30.842487 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.843623 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.845739 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.846448 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.846559 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:30.847218 (+ 659us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:47:30.847361 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:47:30.847452 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:30.847600 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 20:47:30.847677 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 20:47:30.848389 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:30.840537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38499 (local address 127.0.97.3:36683)
0929 20:47:30.841276 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:30.841297 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:30.841507 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:30.841900 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:30.841905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:30.841919 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:47:30.842319 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:30.842333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.843764 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.843770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:30.845581 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:30.845598 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.846640 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.846647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:30.846850 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:30.846927 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:47:30.847468 (+ 541us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:47:30.847480 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 20:47:30.847573 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 20:47:30.847722 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":369,"threads_started":1}
I20260929 20:47:30.859154 8141 raft_consensus.cc:1261] T 3c07a7c0304240cfa9e204a9a711910e P 5086db73455d42d6ba8ecdae4169f287 [term 1 FOLLOWER]: Refusing update from remote peer 2fb8841de3304be998e952b840b3bc7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:47:30.859766 8574 consensus_queue.cc:1035] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5086db73455d42d6ba8ecdae4169f287" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:47:30.862439 8284 raft_consensus.cc:1261] T 3c07a7c0304240cfa9e204a9a711910e P f37db28cb1064e75ac5f7ec8c6d6f677 [term 1 FOLLOWER]: Refusing update from remote peer 2fb8841de3304be998e952b840b3bc7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:47:30.863935 8574 consensus_queue.cc:1035] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f37db28cb1064e75ac5f7ec8c6d6f677" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 20:47:30.905540 8073 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.97.2:45097, user_credentials={real_user=slave}} blocked reactor thread for 43931.4us
I20260929 20:47:30.950407 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:30.938462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60465 (local address 127.0.97.3:44777)
0929 20:47:30.938625 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:30.938631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:30.943621 (+ 4990us) server_negotiation.cc:407] Connection header received
0929 20:47:30.943916 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:30.943923 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:30.944012 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:30.944089 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:47:30.944814 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.945960 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.948219 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.948951 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.949085 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:30.949571 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:47:30.949716 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:47:30.949784 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:30.950005 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 20:47:30.950105 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 20:47:30.951680 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:30.938208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44777 (local address 127.0.97.1:60465)
0929 20:47:30.943513 (+ 5305us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:30.943537 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:30.943738 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:30.944303 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:30.944308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:30.944320 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:47:30.944611 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:30.944624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.946218 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.946224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:30.948053 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:30.948068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:30.949147 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:30.949155 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:30.949330 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:30.949380 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:47:30.949865 (+ 485us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:47:30.949878 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 20:47:30.951138 (+ 1260us) client_negotiation.cc:240] Negotiation successful
0929 20:47:30.951284 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5123,"thread_start_us":4948,"threads_started":1}
I20260929 20:47:30.988991 7983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:47:30.989233 7983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:47:30.992334 7983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a4466d3b70f94c64848db222c5423b10: Sending DeleteTablet for 3 replicas of tablet 3c07a7c0304240cfa9e204a9a711910e
I20260929 20:47:30.993490 8260 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c07a7c0304240cfa9e204a9a711910e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52660
I20260929 20:47:30.993749 8517 tablet_replica.cc:330] T 3c07a7c0304240cfa9e204a9a711910e P f37db28cb1064e75ac5f7ec8c6d6f677: stopping tablet replica
I20260929 20:47:30.993937 8517 raft_consensus.cc:2229] T 3c07a7c0304240cfa9e204a9a711910e P f37db28cb1064e75ac5f7ec8c6d6f677 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:47:30.994109 8517 raft_consensus.cc:2258] T 3c07a7c0304240cfa9e204a9a711910e P f37db28cb1064e75ac5f7ec8c6d6f677 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:30.994407 8121 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c07a7c0304240cfa9e204a9a711910e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48286
I20260929 20:47:30.994693 8516 tablet_replica.cc:330] T 3c07a7c0304240cfa9e204a9a711910e P 5086db73455d42d6ba8ecdae4169f287: stopping tablet replica
I20260929 20:47:30.994908 8516 raft_consensus.cc:2229] T 3c07a7c0304240cfa9e204a9a711910e P 5086db73455d42d6ba8ecdae4169f287 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:47:30.995101 8516 raft_consensus.cc:2258] T 3c07a7c0304240cfa9e204a9a711910e P 5086db73455d42d6ba8ecdae4169f287 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:30.995610 8517 ts_tablet_manager.cc:1905] T 3c07a7c0304240cfa9e204a9a711910e P f37db28cb1064e75ac5f7ec8c6d6f677: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:30.996315 8516 ts_tablet_manager.cc:1905] T 3c07a7c0304240cfa9e204a9a711910e P 5086db73455d42d6ba8ecdae4169f287: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:31.002033 8399 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c07a7c0304240cfa9e204a9a711910e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:47:30 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48414
I20260929 20:47:31.002363 8518 tablet_replica.cc:330] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a: stopping tablet replica
I20260929 20:47:31.002601 8518 raft_consensus.cc:2229] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:47:31.002935 8518 raft_consensus.cc:2258] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:47:31.003093 8516 ts_tablet_manager.cc:1918] T 3c07a7c0304240cfa9e204a9a711910e P 5086db73455d42d6ba8ecdae4169f287: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:47:31.003224 8517 ts_tablet_manager.cc:1918] T 3c07a7c0304240cfa9e204a9a711910e P f37db28cb1064e75ac5f7ec8c6d6f677: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:47:31.003366 8517 log.cc:1196] T 3c07a7c0304240cfa9e204a9a711910e P f37db28cb1064e75ac5f7ec8c6d6f677: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/3c07a7c0304240cfa9e204a9a711910e
I20260929 20:47:31.003372 8516 log.cc:1196] T 3c07a7c0304240cfa9e204a9a711910e P 5086db73455d42d6ba8ecdae4169f287: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/3c07a7c0304240cfa9e204a9a711910e
I20260929 20:47:31.003749 8517 ts_tablet_manager.cc:1939] T 3c07a7c0304240cfa9e204a9a711910e P f37db28cb1064e75ac5f7ec8c6d6f677: Deleting consensus metadata
I20260929 20:47:31.003764 8516 ts_tablet_manager.cc:1939] T 3c07a7c0304240cfa9e204a9a711910e P 5086db73455d42d6ba8ecdae4169f287: Deleting consensus metadata
I20260929 20:47:31.004396 8518 ts_tablet_manager.cc:1905] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:47:31.004750 7970 catalog_manager.cc:4841] TS f37db28cb1064e75ac5f7ec8c6d6f677 (127.0.97.2:45097): tablet 3c07a7c0304240cfa9e204a9a711910e (table test-table [id=f423e425820744129dc2bbb7df97c2d9]) successfully deleted
I20260929 20:47:31.005146 7969 catalog_manager.cc:4841] TS 5086db73455d42d6ba8ecdae4169f287 (127.0.97.1:38499): tablet 3c07a7c0304240cfa9e204a9a711910e (table test-table [id=f423e425820744129dc2bbb7df97c2d9]) successfully deleted
I20260929 20:47:31.007352 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 8055
I20260929 20:47:31.011840 8518 ts_tablet_manager.cc:1918] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:47:31.012027 8518 log.cc:1196] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/3c07a7c0304240cfa9e204a9a711910e
I20260929 20:47:31.012499 8518 ts_tablet_manager.cc:1939] T 3c07a7c0304240cfa9e204a9a711910e P 2fb8841de3304be998e952b840b3bc7a: Deleting consensus metadata
I20260929 20:47:31.013736 7971 catalog_manager.cc:4841] TS 2fb8841de3304be998e952b840b3bc7a (127.0.97.3:44777): tablet 3c07a7c0304240cfa9e204a9a711910e (table test-table [id=f423e425820744129dc2bbb7df97c2d9]) successfully deleted
Sep 29 20:47:31 dist-test-slave-blv7 krb5kdc[7939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260929 20:47:31.034524 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:31.009243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48310 (local address 127.0.97.1:38499)
0929 20:47:31.009439 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:31.009445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:31.014496 (+ 5051us) server_negotiation.cc:407] Connection header received
0929 20:47:31.014793 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:31.014799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:31.014886 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:31.014965 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:31.016740 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:31.017824 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:31.019750 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:31.020498 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:31.020605 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:31.026191 (+ 5586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:31.026211 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:31.026232 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:31.026278 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:31.030751 (+ 4473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:31.031770 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:31.031782 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:31.031786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:31.031987 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:31.032481 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:31.032486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:31.032490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:31.032968 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:31.033293 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:31.033876 (+ 583us) server_negotiation.cc:299] Negotiation successful
0929 20:47:31.034128 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:47:31.036132 8186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:31.184441 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 8055
I20260929 20:47:31.204671 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 8195
Sep 29 20:47:31 dist-test-slave-blv7 krb5kdc[7939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260929 20:47:31.218060 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:31.205344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52678 (local address 127.0.97.2:45097)
0929 20:47:31.205782 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:31.205786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:31.205802 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:47:31.205854 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:31.205858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:31.205914 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:31.206001 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:31.207092 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:31.207816 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:31.209000 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:31.209474 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:31.209573 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:31.213158 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:31.213171 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:31.213186 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:31.213221 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:31.215709 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:31.216289 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:31.216296 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:31.216299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:31.216421 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:31.216802 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:31.216805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:31.216807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:31.217137 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:31.217359 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:31.217584 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 20:47:31.217735 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":239,"threads_started":1}
I20260929 20:47:31.218734 8325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:31.333699 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 8195
I20260929 20:47:31.350487 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 8334
Sep 29 20:47:31 dist-test-slave-blv7 krb5kdc[7939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260929 20:47:31.364559 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:31.351298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.0.97.3:44777)
0929 20:47:31.351547 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:31.351552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:31.351572 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:47:31.351638 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:31.351643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:31.351712 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:31.351805 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:31.352999 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:31.353676 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:31.354877 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:31.355306 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:31.355413 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:31.359551 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:31.359565 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:31.359582 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:31.359620 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:31.362007 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:31.362639 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:31.362648 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:31.362650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:31.362765 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:31.363126 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:31.363131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:31.363134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:31.363491 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:31.363784 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:31.364058 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 20:47:31.364278 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 20:47:31.365151 8464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:31.483068 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 8334
I20260929 20:47:31.500275 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 7952
I20260929 20:47:31.501415 8024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:31.611228 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 7952
2026-09-29T20:47:31Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5329 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:47:31 dist-test-slave-blv7 krb5kdc[8602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:47:31 dist-test-slave-blv7 krb5kdc[8602](info): set up 2 sockets
Sep 29 20:47:31 dist-test-slave-blv7 krb5kdc[8602](info): commencing operation
krb5kdc: starting...
W20260929 20:47:32.615659 388 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:47:32 dist-test-slave-blv7 krb5kdc[8602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:47:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:47:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:47:32.814834 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33761
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:33.121665 8615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:33.121932 8615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:33.122061 8615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:33.129302 8615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:47:33.129427 8615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:33.129503 8615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:33.129590 8615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:47:33.129655 8615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:47:33.139423 8615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:33.140640 8615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:33.141922 8615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:33.152750 8620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:33.152810 8623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:33.152835 8621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:33.153550 8615 server_base.cc:1034] running on GCE node
I20260929 20:47:33.154294 8615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:33.155781 8615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:33.157008 8615 hybrid_clock.cc:648] HybridClock initialized: now 1790714853156947 us; error 49 us; skew 500 ppm
I20260929 20:47:33.157347 8615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T20:47:33Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1622 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:47:33 dist-test-slave-blv7 krb5kdc[8629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:47:33 dist-test-slave-blv7 krb5kdc[8629](info): set up 2 sockets
Sep 29 20:47:33 dist-test-slave-blv7 krb5kdc[8629](info): commencing operation
krb5kdc: starting...
W20260929 20:47:34.235632 388 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:47:34 dist-test-slave-blv7 krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:47:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:47:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:47:34.432415 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33411
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:34.736414 8642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:34.736711 8642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:34.736882 8642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:34.744025 8642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:47:34.744140 8642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:34.744225 8642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:34.744303 8642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:47:34.744374 8642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:47:34.753834 8642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:34.755050 8642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:34.756356 8642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:34.767864 8648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:34.767964 8647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:34.767994 8650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:34.768680 8642 server_base.cc:1034] running on GCE node
I20260929 20:47:34.769358 8642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:34.770649 8642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:34.771848 8642 hybrid_clock.cc:648] HybridClock initialized: now 1790714854771799 us; error 42 us; skew 500 ppm
I20260929 20:47:34.772145 8642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:34 dist-test-slave-blv7 krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:34.775043 8642 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:47:34.776443 8642 webserver.cc:460] Webserver started at http://127.0.97.62:37329/ using document root <none> and password file <none>
I20260929 20:47:34.777071 8642 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:34.777180 8642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:34.777452 8642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:34.780197 8642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "c5b2ff87cba74d9bb249338ddcb0dab9"
format_stamp: "Formatted at 2026-09-29 20:47:34 on dist-test-slave-blv7"
server_key: "c261d680278af64e0e5aca27585b4efc"
server_key_iv: "9f626d53d58ec995218d2bf7c10ad6e0"
server_key_version: "encryptionkey@0"
I20260929 20:47:34.780934 8642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "c5b2ff87cba74d9bb249338ddcb0dab9"
format_stamp: "Formatted at 2026-09-29 20:47:34 on dist-test-slave-blv7"
server_key: "c261d680278af64e0e5aca27585b4efc"
server_key_iv: "9f626d53d58ec995218d2bf7c10ad6e0"
server_key_version: "encryptionkey@0"
I20260929 20:47:34.785532 8642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 20:47:34.788661 8657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:34.789863 8642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 20:47:34.790010 8642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "c5b2ff87cba74d9bb249338ddcb0dab9"
format_stamp: "Formatted at 2026-09-29 20:47:34 on dist-test-slave-blv7"
server_key: "c261d680278af64e0e5aca27585b4efc"
server_key_iv: "9f626d53d58ec995218d2bf7c10ad6e0"
server_key_version: "encryptionkey@0"
I20260929 20:47:34.790163 8642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:34.818529 8642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:34.822633 8642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:34.822888 8642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:34.839540 8642 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33411
I20260929 20:47:34.839561 8708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33411 every 8 connection(s)
I20260929 20:47:34.840811 8642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:47:34.844331 8709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:34.846007 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 8642
I20260929 20:47:34.846175 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:47:34.846808 388 external_mini_cluster.cc:1428] Setting key e84bfcaa0da0dc642470e00d727164d6
I20260929 20:47:34.855227 8709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9: Bootstrap starting.
I20260929 20:47:34.858721 8709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:34.859958 8709 log.cc:824] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:47:34 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:47:34.862778 8709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9: No bootstrap required, opened a new log
I20260929 20:47:34.868117 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:34.848675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39080 (local address 127.0.97.62:33411)
0929 20:47:34.849418 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:34.849429 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:34.849470 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:47:34.851009 (+ 1539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:34.851071 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:34.851449 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:34.851761 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:34.853382 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:34.854606 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:34.856239 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:34.856901 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:34.857024 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:34.861465 (+ 4441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:34.861472 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:34.861486 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:34.861522 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:34.864413 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:34.865031 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:34.865037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:34.865043 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:34.865204 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:34.865547 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:34.865551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:34.865553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:34.866052 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:34.866351 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:34.866732 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 20:47:34.867149 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":319,"threads_started":1}
I20260929 20:47:34.870112 8709 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b2ff87cba74d9bb249338ddcb0dab9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33411 } }
I20260929 20:47:34.870435 8709 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:34.870519 8709 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b2ff87cba74d9bb249338ddcb0dab9, State: Initialized, Role: FOLLOWER
I20260929 20:47:34.871037 8709 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b2ff87cba74d9bb249338ddcb0dab9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33411 } }
I20260929 20:47:34.871253 8709 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:47:34.871371 8709 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:47:34.871510 8709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:34.873412 8709 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b2ff87cba74d9bb249338ddcb0dab9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33411 } }
I20260929 20:47:34.873850 8709 leader_election.cc:302] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5b2ff87cba74d9bb249338ddcb0dab9; no voters:
I20260929 20:47:34.874320 8709 leader_election.cc:288] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:47:34.874434 8714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:47:34.875031 8714 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [term 1 LEADER]: Becoming Leader. State: Replica: c5b2ff87cba74d9bb249338ddcb0dab9, State: Running, Role: LEADER
I20260929 20:47:34.875556 8714 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b2ff87cba74d9bb249338ddcb0dab9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33411 } }
I20260929 20:47:34.876048 8709 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:47:34.877997 8716 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5b2ff87cba74d9bb249338ddcb0dab9. Latest consensus state: current_term: 1 leader_uuid: "c5b2ff87cba74d9bb249338ddcb0dab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b2ff87cba74d9bb249338ddcb0dab9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33411 } } }
I20260929 20:47:34.877955 8715 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5b2ff87cba74d9bb249338ddcb0dab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b2ff87cba74d9bb249338ddcb0dab9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33411 } } }
I20260929 20:47:34.878423 8715 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:34.879107 8716 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:34.880239 8722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:47:34.885912 8722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:47:34.895828 8722 catalog_manager.cc:1347] Generated new cluster ID: e0a16769dc47474da49f96f46f76e532
I20260929 20:47:34.895927 8722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:47:34.906507 8722 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:47:34.907752 8722 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:47:34.919837 8722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9: Generated new TSK 0
I20260929 20:47:34.920651 8722 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:47:34.978492 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33411
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:35.284719 8735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:35.285002 8735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:35.285120 8735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:35.292400 8735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:35.292510 8735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:35.292652 8735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:47:35.302386 8735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:35.303604 8735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:35.304840 8735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:35.317360 8740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:35.317384 8741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:35.317420 8743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:35.318112 8735 server_base.cc:1034] running on GCE node
I20260929 20:47:35.318657 8735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:35.319629 8735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:35.320868 8735 hybrid_clock.cc:648] HybridClock initialized: now 1790714855320783 us; error 64 us; skew 500 ppm
I20260929 20:47:35.321228 8735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:35 dist-test-slave-blv7 krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:35.324301 8735 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:47:35.325726 8735 webserver.cc:460] Webserver started at http://127.0.97.1:40511/ using document root <none> and password file <none>
I20260929 20:47:35.326370 8735 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:35.326478 8735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:35.326751 8735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:35.329662 8735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "7c2e507b59db4da48967409242d56e4f"
format_stamp: "Formatted at 2026-09-29 20:47:35 on dist-test-slave-blv7"
server_key: "a992b6c6881861adfdecd101d95f702a"
server_key_iv: "37eac6c96ca2bfaef418deef2e847060"
server_key_version: "encryptionkey@0"
I20260929 20:47:35.330425 8735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7c2e507b59db4da48967409242d56e4f"
format_stamp: "Formatted at 2026-09-29 20:47:35 on dist-test-slave-blv7"
server_key: "a992b6c6881861adfdecd101d95f702a"
server_key_iv: "37eac6c96ca2bfaef418deef2e847060"
server_key_version: "encryptionkey@0"
I20260929 20:47:35.335279 8735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 20:47:35.338635 8750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:35.340036 8735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:47:35.340194 8735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "7c2e507b59db4da48967409242d56e4f"
format_stamp: "Formatted at 2026-09-29 20:47:35 on dist-test-slave-blv7"
server_key: "a992b6c6881861adfdecd101d95f702a"
server_key_iv: "37eac6c96ca2bfaef418deef2e847060"
server_key_version: "encryptionkey@0"
I20260929 20:47:35.340373 8735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:35.352763 8735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:35.356679 8735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:35.356925 8735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:35.357806 8735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:35.359100 8735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:35.359176 8735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:35.359290 8735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:35.359340 8735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:35.388113 8735 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37587
I20260929 20:47:35.388155 8862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37587 every 8 connection(s)
I20260929 20:47:35.389285 8735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:47:35.396750 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 8735
I20260929 20:47:35.396944 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:47:35.397624 388 external_mini_cluster.cc:1428] Setting key 83b89ceca2324b87d7c6fb2bf3755a00
Sep 29 20:47:35 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:47:35.409125 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:35.391515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48641 (local address 127.0.97.62:33411)
0929 20:47:35.391916 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:35.391921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:35.392791 (+ 870us) server_negotiation.cc:407] Connection header received
0929 20:47:35.394629 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:35.394635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:35.394712 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:35.394805 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:35.396268 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:35.397302 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:35.399051 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:35.399610 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:35.399733 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:35.403038 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:35.403044 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:35.403048 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:35.403094 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:35.406029 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:35.406920 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:35.406924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:35.406926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:35.407067 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:35.407481 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:35.407485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:35.407488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:35.407815 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:35.407938 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:35.408597 (+ 659us) server_negotiation.cc:299] Negotiation successful
0929 20:47:35.408812 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":183,"threads_started":1}
I20260929 20:47:35.410082 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:35.391988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.1:48641)
0929 20:47:35.392640 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:35.392681 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:35.394369 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:35.395066 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:35.395077 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:35.395561 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:35.396112 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:35.396127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:35.397454 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:35.397458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:35.398895 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:35.398908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:35.399809 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:35.399814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:35.400007 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:35.400606 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:35.400627 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:35.402815 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:35.406169 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:35.406180 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:35.406197 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:35.406803 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:35.407171 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:35.407175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:35.407177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:35.407376 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:35.407963 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:35.407969 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:35.408281 (+ 312us) client_negotiation.cc:769] Sending connection context
0929 20:47:35.408504 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 20:47:35.408898 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":245,"threads_started":1}
I20260929 20:47:35.411512 8863 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33411
I20260929 20:47:35.411844 8863 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:35.412636 8863 heartbeater.cc:511] Master 127.0.97.62:33411 requested a full tablet report, sending...
I20260929 20:47:35.414703 8674 ts_manager.cc:194] Registered new tserver with Master: 7c2e507b59db4da48967409242d56e4f (127.0.97.1:37587)
I20260929 20:47:35.416158 8674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48641
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:47:35.443954 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33411
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:35.756026 8869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:35.756309 8869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:35.756412 8869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:35.763899 8869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:35.764012 8869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:35.764142 8869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:47:35.773811 8869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:35.774956 8869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:35.776266 8869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:35.789084 8875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:35.789045 8877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:35.789139 8874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:35.789793 8869 server_base.cc:1034] running on GCE node
I20260929 20:47:35.790380 8869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:35.791375 8869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:35.792582 8869 hybrid_clock.cc:648] HybridClock initialized: now 1790714855792528 us; error 53 us; skew 500 ppm
I20260929 20:47:35.792922 8869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:35 dist-test-slave-blv7 krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:35.795892 8869 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:47:35.797199 8869 webserver.cc:460] Webserver started at http://127.0.97.2:46641/ using document root <none> and password file <none>
I20260929 20:47:35.797821 8869 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:35.797931 8869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:35.798202 8869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:35.800927 8869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "4ebbec2caa34447383ff557cb2bed4b7"
format_stamp: "Formatted at 2026-09-29 20:47:35 on dist-test-slave-blv7"
server_key: "7c2859b06ed54998d871960285d0a434"
server_key_iv: "e549bd2e7b49473346d3e11afe5079ea"
server_key_version: "encryptionkey@0"
I20260929 20:47:35.801637 8869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4ebbec2caa34447383ff557cb2bed4b7"
format_stamp: "Formatted at 2026-09-29 20:47:35 on dist-test-slave-blv7"
server_key: "7c2859b06ed54998d871960285d0a434"
server_key_iv: "e549bd2e7b49473346d3e11afe5079ea"
server_key_version: "encryptionkey@0"
I20260929 20:47:35.806352 8869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260929 20:47:35.809628 8884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:35.810954 8869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:47:35.811120 8869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "4ebbec2caa34447383ff557cb2bed4b7"
format_stamp: "Formatted at 2026-09-29 20:47:35 on dist-test-slave-blv7"
server_key: "7c2859b06ed54998d871960285d0a434"
server_key_iv: "e549bd2e7b49473346d3e11afe5079ea"
server_key_version: "encryptionkey@0"
I20260929 20:47:35.811305 8869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:35.844518 8869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:35.848485 8869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:35.848726 8869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:35.849633 8869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:35.850865 8869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:35.850939 8869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:35.851080 8869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:35.851133 8869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:35.879611 8869 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45933
I20260929 20:47:35.879665 8996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45933 every 8 connection(s)
I20260929 20:47:35.880883 8869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:47:35.882665 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 8869
I20260929 20:47:35.882807 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:47:35.883322 388 external_mini_cluster.cc:1428] Setting key 5602739a44ff63b2f25bbc28affa8e1e
Sep 29 20:47:35 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:47:35.901003 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:35.883261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38333 (local address 127.0.97.62:33411)
0929 20:47:35.883466 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:35.883471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:35.884633 (+ 1162us) server_negotiation.cc:407] Connection header received
0929 20:47:35.886938 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:35.886943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:35.887005 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:35.887113 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:35.889078 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:35.889994 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:35.891632 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:35.892188 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:35.892307 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:35.895882 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:35.895887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:35.895890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:35.895919 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:35.898162 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:35.898944 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:35.898948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:35.898950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:35.899112 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:35.899524 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:35.899526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:35.899528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:35.899759 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:35.899856 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:35.900611 (+ 755us) server_negotiation.cc:299] Negotiation successful
0929 20:47:35.900771 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 20:47:35.902448 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:35.883752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.2:38333)
0929 20:47:35.884494 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:35.884536 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:35.886624 (+ 2088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:35.887525 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:35.887537 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:35.888235 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:35.888834 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:35.888852 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:35.890158 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:35.890164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:35.891508 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:35.891520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:35.892370 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:35.892376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:35.892653 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:35.893394 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:35.893419 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:35.895668 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:35.898303 (+ 2635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:35.898311 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:35.898329 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:35.898845 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:35.899198 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:35.899203 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:35.899206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:35.899431 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:35.899862 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:35.899869 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:35.900242 (+ 373us) client_negotiation.cc:769] Sending connection context
0929 20:47:35.900551 (+ 309us) client_negotiation.cc:240] Negotiation successful
0929 20:47:35.901062 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":305,"threads_started":1}
I20260929 20:47:35.903729 8997 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33411
I20260929 20:47:35.904047 8997 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:47:35.904860 8997 heartbeater.cc:511] Master 127.0.97.62:33411 requested a full tablet report, sending...
I20260929 20:47:35.906095 8674 ts_manager.cc:194] Registered new tserver with Master: 4ebbec2caa34447383ff557cb2bed4b7 (127.0.97.2:45933)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 20:47:35.906955 8674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:47:35.930305 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33411
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:36.234278 9002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:36.234565 9002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:36.234710 9002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:36.241912 9002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:36.242036 9002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:36.242218 9002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:47:36.251837 9002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:36.253034 9002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:36.254324 9002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:36.266587 9007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:36.266587 9010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:36.266674 9008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:36.272423 9002 server_base.cc:1034] running on GCE node
I20260929 20:47:36.273010 9002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:36.273921 9002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:36.275107 9002 hybrid_clock.cc:648] HybridClock initialized: now 1790714856275058 us; error 43 us; skew 500 ppm
I20260929 20:47:36.275411 9002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:36 dist-test-slave-blv7 krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:36.278446 9002 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:47:36.279891 9002 webserver.cc:460] Webserver started at http://127.0.97.3:35107/ using document root <none> and password file <none>
I20260929 20:47:36.280577 9002 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:36.280685 9002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:36.280975 9002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:36.283785 9002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "5ba6a2566cbb4c60b6139d8ec820eceb"
format_stamp: "Formatted at 2026-09-29 20:47:36 on dist-test-slave-blv7"
server_key: "d6cc36095f12a937eb32fa21c5bcca0a"
server_key_iv: "b2ab2c8c6829f18ff506140591d69f20"
server_key_version: "encryptionkey@0"
I20260929 20:47:36.284509 9002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5ba6a2566cbb4c60b6139d8ec820eceb"
format_stamp: "Formatted at 2026-09-29 20:47:36 on dist-test-slave-blv7"
server_key: "d6cc36095f12a937eb32fa21c5bcca0a"
server_key_iv: "b2ab2c8c6829f18ff506140591d69f20"
server_key_version: "encryptionkey@0"
I20260929 20:47:36.289219 9002 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:47:36.292392 9017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:36.293715 9002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 20:47:36.293860 9002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "5ba6a2566cbb4c60b6139d8ec820eceb"
format_stamp: "Formatted at 2026-09-29 20:47:36 on dist-test-slave-blv7"
server_key: "d6cc36095f12a937eb32fa21c5bcca0a"
server_key_iv: "b2ab2c8c6829f18ff506140591d69f20"
server_key_version: "encryptionkey@0"
I20260929 20:47:36.294044 9002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:36.314633 9002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:36.318830 9002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:36.319106 9002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:36.319998 9002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:36.321267 9002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:36.321341 9002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:36.321444 9002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:36.321489 9002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:36.348747 9002 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39723
I20260929 20:47:36.348793 9129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39723 every 8 connection(s)
I20260929 20:47:36.349900 9002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:47:36.358047 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 9002
I20260929 20:47:36.358174 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:47:36.358560 388 external_mini_cluster.cc:1428] Setting key fce61c237538831dc118d00bef96e020
Sep 29 20:47:36 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:47:36.368163 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:36.352246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34411 (local address 127.0.97.62:33411)
0929 20:47:36.352425 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:36.352429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:36.353530 (+ 1101us) server_negotiation.cc:407] Connection header received
0929 20:47:36.355281 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:36.355295 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:36.355373 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:36.355459 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:36.356851 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:36.357572 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:36.359228 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:36.359799 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:36.359922 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:36.363255 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:36.363260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:36.363263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:36.363294 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:36.365662 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:36.366287 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:36.366291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:36.366294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:36.366407 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:36.366750 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:36.366753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:36.366755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:36.367000 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:36.367151 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:36.367757 (+ 606us) server_negotiation.cc:299] Negotiation successful
0929 20:47:36.367909 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:47:36.369024 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:36.352731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.3:34411)
0929 20:47:36.353388 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:36.353426 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:36.355006 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:36.355695 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:36.355703 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:36.356161 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:36.356682 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:36.356698 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:36.357722 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:36.357726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:36.359085 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:36.359100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:36.359984 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:36.359989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:36.360200 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:36.360923 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:36.360949 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:36.363078 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:36.365775 (+ 2697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:36.365781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:36.365803 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:36.366170 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:36.366486 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:47:36.366489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:47:36.366491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:47:36.366647 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:47:36.367135 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:36.367140 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:36.367410 (+ 270us) client_negotiation.cc:769] Sending connection context
0929 20:47:36.367614 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 20:47:36.367962 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":258,"threads_started":1}
I20260929 20:47:36.370266 9130 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33411
I20260929 20:47:36.370532 9130 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:36.371196 9130 heartbeater.cc:511] Master 127.0.97.62:33411 requested a full tablet report, sending...
I20260929 20:47:36.372591 8673 ts_manager.cc:194] Registered new tserver with Master: 5ba6a2566cbb4c60b6139d8ec820eceb (127.0.97.3:39723)
I20260929 20:47:36.373453 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:47:36.373677 8673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34411
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260929 20:47:36.419390 8863 heartbeater.cc:503] Master 127.0.97.62:33411 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 20:47:36.495158 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 8735
Sep 29 20:47:36 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:47:36.510100 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:36.496324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.0.97.1:37587)
0929 20:47:36.496883 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:36.496891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:36.496917 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:47:36.496977 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:36.496984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:36.497201 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:36.497346 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:36.498658 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:36.499715 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:36.500843 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:36.501394 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:36.501489 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:36.505164 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:36.505174 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:36.505188 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:36.505221 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:36.507843 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:36.508411 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:36.508418 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:36.508420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:36.508537 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:36.508856 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:36.508858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:36.508861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:36.509158 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:36.509371 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:36.509581 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 20:47:36.509761 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":282,"threads_started":1}
I20260929 20:47:36.511436 8858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:36.592833 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 8735
I20260929 20:47:36.606101 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 8869
Sep 29 20:47:36 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:47:36.620800 9142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:36.606939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.0.97.2:45933)
0929 20:47:36.607455 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:36.607462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:36.607484 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:47:36.607544 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:36.607550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:36.607742 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:36.607886 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:36.609085 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:36.610064 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:36.611386 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:36.611903 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:36.612012 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:36.615814 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:36.615824 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:36.615837 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:36.615867 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:36.618277 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:36.618895 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:36.618901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:36.618904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:36.619040 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:36.619362 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:36.619367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:36.619369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:36.619675 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:36.619935 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:36.620242 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 20:47:36.620475 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":290,"threads_started":1}
I20260929 20:47:36.622046 8992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:36.701170 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 8869
I20260929 20:47:36.713186 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 9002
Sep 29 20:47:36 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:47:36.728741 9144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:36.714010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46840 (local address 127.0.97.3:39723)
0929 20:47:36.714586 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:36.714594 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:36.714622 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:47:36.714684 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:36.714690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:36.714895 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:36.715052 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:36.716353 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:36.717422 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:36.718694 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:36.719394 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:36.719491 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:36.723564 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:36.723575 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:36.723590 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:36.723622 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:36.726096 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:36.726667 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:36.726675 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:36.726677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:36.726795 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:36.727233 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:36.727237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:36.727239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:36.727589 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:36.727843 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:36.728182 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 20:47:36.728443 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":282,"threads_started":1}
I20260929 20:47:36.729947 9125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:36.810161 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 9002
I20260929 20:47:36.822451 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 8642
I20260929 20:47:36.823606 8704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:36.922931 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 8642
I20260929 20:47:36.938843 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33411
--webserver_interface=127.0.97.62
--webserver_port=37329
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:37.239971 9147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:37.240458 9147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:37.240620 9147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:37.247740 9147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:47:37.247839 9147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:37.247910 9147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:37.247983 9147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:47:37.248039 9147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:47:37.257653 9147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=37329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:37.258960 9147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:37.260232 9147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:37.271274 9152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:37.271239 9155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:37.271230 9153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:37.272106 9147 server_base.cc:1034] running on GCE node
I20260929 20:47:37.272873 9147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:37.274152 9147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:37.275352 9147 hybrid_clock.cc:648] HybridClock initialized: now 1790714857275300 us; error 45 us; skew 500 ppm
I20260929 20:47:37.275656 9147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:37 dist-test-slave-blv7 krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:37.278422 9147 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:47:37.279824 9147 webserver.cc:460] Webserver started at http://127.0.97.62:37329/ using document root <none> and password file <none>
I20260929 20:47:37.280442 9147 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:37.280560 9147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:37.285598 9147 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 20:47:37.288265 9162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:37.289530 9147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:47:37.289680 9147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "c5b2ff87cba74d9bb249338ddcb0dab9"
format_stamp: "Formatted at 2026-09-29 20:47:34 on dist-test-slave-blv7"
server_key: "c261d680278af64e0e5aca27585b4efc"
server_key_iv: "9f626d53d58ec995218d2bf7c10ad6e0"
server_key_version: "encryptionkey@0"
I20260929 20:47:37.290601 9147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:37.323068 9147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:37.326829 9147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:37.327087 9147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:37.343609 9147 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33411
I20260929 20:47:37.343647 9213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33411 every 8 connection(s)
I20260929 20:47:37.345022 9147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:47:37.345909 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 9147
I20260929 20:47:37.346693 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:37587
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:33411
--webserver_port=40511
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:47:37.352010 9214 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:47:37.355326 9214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9: Bootstrap starting.
I20260929 20:47:37.373252 9214 log.cc:824] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:37.381494 9214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:47:37.381970 9214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9: Bootstrap complete.
I20260929 20:47:37.390482 9214 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b2ff87cba74d9bb249338ddcb0dab9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33411 } }
I20260929 20:47:37.391316 9214 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b2ff87cba74d9bb249338ddcb0dab9, State: Initialized, Role: FOLLOWER
I20260929 20:47:37.391944 9214 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b2ff87cba74d9bb249338ddcb0dab9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33411 } }
I20260929 20:47:37.392148 9214 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:47:37.392251 9214 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:47:37.392409 9214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:47:37.396222 9214 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b2ff87cba74d9bb249338ddcb0dab9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33411 } }
I20260929 20:47:37.396652 9214 leader_election.cc:302] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5b2ff87cba74d9bb249338ddcb0dab9; no voters:
I20260929 20:47:37.397184 9214 leader_election.cc:288] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 20:47:37.397362 9218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:47:37.398772 9214 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:47:37.403191 9218 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [term 2 LEADER]: Becoming Leader. State: Replica: c5b2ff87cba74d9bb249338ddcb0dab9, State: Running, Role: LEADER
I20260929 20:47:37.403697 9218 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b2ff87cba74d9bb249338ddcb0dab9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33411 } }
I20260929 20:47:37.406116 9219 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c5b2ff87cba74d9bb249338ddcb0dab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b2ff87cba74d9bb249338ddcb0dab9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33411 } } }
I20260929 20:47:37.406050 9220 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5b2ff87cba74d9bb249338ddcb0dab9. Latest consensus state: current_term: 2 leader_uuid: "c5b2ff87cba74d9bb249338ddcb0dab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b2ff87cba74d9bb249338ddcb0dab9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33411 } } }
I20260929 20:47:37.406524 9219 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:37.406530 9220 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9 [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:37.407953 9233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:47:37.413106 9233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:47:37.414404 9233 catalog_manager.cc:1259] Loaded cluster ID: e0a16769dc47474da49f96f46f76e532
I20260929 20:47:37.414526 9233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:47:37.418217 9233 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:47:37.419589 9233 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c5b2ff87cba74d9bb249338ddcb0dab9: Loaded TSK: 0
I20260929 20:47:37.421048 9233 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 20:47:37.656594 9216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:37.656842 9216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:37.656958 9216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:37.664112 9216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:37.664211 9216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:37.664358 9216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:47:37.673719 9216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:37587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=40511
--tserver_master_addrs=127.0.97.62:33411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:37.674979 9216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:37.676304 9216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:37.690248 9240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:37.690321 9241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:37.690644 9243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:37.691578 9216 server_base.cc:1034] running on GCE node
I20260929 20:47:37.692154 9216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:37.693049 9216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:37.694296 9216 hybrid_clock.cc:648] HybridClock initialized: now 1790714857694235 us; error 51 us; skew 500 ppm
I20260929 20:47:37.694664 9216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:37 dist-test-slave-blv7 krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714857, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:37.698559 9216 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 20:47:37.700330 9216 webserver.cc:460] Webserver started at http://127.0.97.1:40511/ using document root <none> and password file <none>
I20260929 20:47:37.701241 9216 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:37.701380 9216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:37.708204 9216 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:47:37.711354 9250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:37.712836 9216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:47:37.712982 9216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "7c2e507b59db4da48967409242d56e4f"
format_stamp: "Formatted at 2026-09-29 20:47:35 on dist-test-slave-blv7"
server_key: "a992b6c6881861adfdecd101d95f702a"
server_key_iv: "37eac6c96ca2bfaef418deef2e847060"
server_key_version: "encryptionkey@0"
I20260929 20:47:37.714177 9216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:37.758731 9216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:37.762521 9216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:37.762744 9216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:37.763728 9216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:37.764993 9216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:37.765084 9216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:37.765157 9216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:37.765214 9216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:37.793524 9216 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37587
I20260929 20:47:37.793608 9362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37587 every 8 connection(s)
I20260929 20:47:37.794934 9216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:47:37.795917 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 9216
I20260929 20:47:37.796610 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45933
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:33411
--webserver_port=46641
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:47:37 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:37 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:37.816284 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:37.798086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.1:58819)
0929 20:47:37.798886 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:37.798931 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:37.800784 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:37.804598 (+ 3814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:37.804609 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:37.805278 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:37.805984 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:37.806008 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:37.807572 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:37.807577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:37.809323 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:37.809336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:37.810288 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:37.810294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:37.810534 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:37.811237 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:37.811263 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:37.814935 (+ 3672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":446,"threads_started":1}
W20260929 20:47:37.816439 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:37.799083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58819 (local address 127.0.97.62:33411)
0929 20:47:37.803397 (+ 4314us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:37.803409 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:37.803448 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:47:37.803731 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:37.803744 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:37.804078 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:37.804436 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:37.806173 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:37.807428 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:37.809449 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:37.810092 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:37.810218 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:37.815400 (+ 5182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58819: BlockingRecv error: recv got EOF from 127.0.97.1:58819 (error 108)
Metrics: {"server-negotiator.queue_time_us":4107,"thread_start_us":3929,"threads_started":1}
W20260929 20:47:37.817232 9363 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33411 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:33411: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 29 20:47:37 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:37 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:37.826134 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:37.817760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.1:48647)
0929 20:47:37.817935 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:37.817956 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:37.818069 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:37.818389 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:37.818394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:37.818810 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:37.818994 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:37.819006 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:37.820052 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:37.820058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:37.821534 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:37.821548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:37.822179 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:37.822185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:37.822352 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:37.822937 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:37.822959 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:37.825868 (+ 2909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 20:47:37.826418 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:37.817879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48647 (local address 127.0.97.62:33411)
0929 20:47:37.818006 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:37.818010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:37.818026 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:47:37.818200 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:37.818203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:37.818262 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:37.818336 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:37.819178 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:37.819922 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:37.821667 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:37.822059 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:37.822147 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:37.826091 (+ 3944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48647: BlockingRecv error: recv got EOF from 127.0.97.1:48647 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 29 20:47:37 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:37 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:37.835644 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:37.826768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.1:58855)
0929 20:47:37.826950 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:37.826970 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:37.827167 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:37.827483 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:37.827489 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:37.827890 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:37.828091 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:37.828125 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:37.829053 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:37.829059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:37.830661 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:37.830678 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:37.831384 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:37.831391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:37.831581 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:37.832176 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:37.832201 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:37.835326 (+ 3125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 20:47:37.835673 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:37.826957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58855 (local address 127.0.97.62:33411)
0929 20:47:37.827111 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:37.827114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:37.827131 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:47:37.827281 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:37.827284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:37.827359 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:37.827434 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:37.828237 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:37.828914 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:37.830793 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:37.831246 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:37.831350 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:37.835488 (+ 4138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58855: BlockingRecv error: recv got EOF from 127.0.97.1:58855 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 20:47:37.836051 9363 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:47:38.119279 9367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:38.119565 9367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:38.119704 9367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:38.126801 9367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:38.126925 9367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:38.127115 9367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:47:38.136469 9367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=46641
--tserver_master_addrs=127.0.97.62:33411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:38.137818 9367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:38.139155 9367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:38.151831 9372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:38.151806 9373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:38.152061 9375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:38.152572 9367 server_base.cc:1034] running on GCE node
I20260929 20:47:38.153189 9367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:38.154090 9367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:38.155272 9367 hybrid_clock.cc:648] HybridClock initialized: now 1790714858155217 us; error 47 us; skew 500 ppm
I20260929 20:47:38.155562 9367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:38 dist-test-slave-blv7 krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714858, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:38.158488 9367 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 20:47:38.159871 9367 webserver.cc:460] Webserver started at http://127.0.97.2:46641/ using document root <none> and password file <none>
I20260929 20:47:38.160527 9367 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:38.160650 9367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:38.165855 9367 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260929 20:47:38.168486 9382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:38.169731 9367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:47:38.169883 9367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "4ebbec2caa34447383ff557cb2bed4b7"
format_stamp: "Formatted at 2026-09-29 20:47:35 on dist-test-slave-blv7"
server_key: "7c2859b06ed54998d871960285d0a434"
server_key_iv: "e549bd2e7b49473346d3e11afe5079ea"
server_key_version: "encryptionkey@0"
I20260929 20:47:38.170749 9367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:38.199512 9367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:38.203585 9367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:38.203840 9367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:38.204761 9367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:38.206018 9367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:38.206121 9367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:38.206223 9367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:38.206290 9367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:38.235877 9367 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45933
I20260929 20:47:38.235951 9494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45933 every 8 connection(s)
I20260929 20:47:38.237243 9367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:47:38.239339 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 9367
I20260929 20:47:38.239956 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39723
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:33411
--webserver_port=35107
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:47:38 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:38 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 20:47:38.253125 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:38.239707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34609 (local address 127.0.97.62:33411)
0929 20:47:38.239911 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:38.239917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:38.241233 (+ 1316us) server_negotiation.cc:407] Connection header received
0929 20:47:38.243501 (+ 2268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:38.243506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:38.243571 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:38.243674 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:38.245472 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.246232 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.247916 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.248333 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.248440 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:38.252896 (+ 4456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34609: BlockingRecv error: recv got EOF from 127.0.97.2:34609 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:47:38.253811 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:38.240229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.2:34609)
0929 20:47:38.241051 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:38.241104 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:38.243208 (+ 2104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:38.243913 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:38.243923 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:38.244541 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:38.245277 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:38.245300 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.246381 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.246386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:38.247770 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:38.247782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.248545 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.248552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:38.248767 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:38.249405 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:38.249427 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:38.252692 (+ 3265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":283,"threads_started":1}
W20260929 20:47:38.254571 9495 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33411 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:33411: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 29 20:47:38 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:38 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:38.261909 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:38.255157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.2:34589)
0929 20:47:38.255299 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:38.255316 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:38.255410 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:38.255673 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:38.255677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:38.255982 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:38.256107 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:38.256115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.257052 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.257057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:38.257918 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:38.257929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.258576 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.258582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:38.258725 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:38.259246 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:38.259263 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:38.261649 (+ 2386us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 20:47:38.262059 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:38.255273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34589 (local address 127.0.97.62:33411)
0929 20:47:38.255426 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:38.255431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:38.255448 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:47:38.255501 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:38.255505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:38.255561 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:38.255635 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:38.256233 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.256928 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.258043 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.258446 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.258546 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:38.261856 (+ 3310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34589: BlockingRecv error: recv got EOF from 127.0.97.2:34589 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 20:47:38 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:38 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:38.269512 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:38.262557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.2:55291)
0929 20:47:38.262700 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:38.262716 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:38.262803 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:38.263129 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:38.263133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:38.263425 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:38.263554 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:38.263561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.264602 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.264608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:38.265504 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:38.265514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.266161 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.266166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:38.266334 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:38.266818 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:38.266835 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:38.269233 (+ 2398us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 20:47:38.269713 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:38.262722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55291 (local address 127.0.97.62:33411)
0929 20:47:38.262894 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:38.262899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:38.262914 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:47:38.262948 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:38.262951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:38.263003 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:38.263108 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:38.263676 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.264484 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.265635 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.266038 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.266139 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:38.269424 (+ 3285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55291: BlockingRecv error: recv got EOF from 127.0.97.2:55291 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 20:47:38.269883 9495 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:47:38.550760 9498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:38.551011 9498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:38.551190 9498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:38.557988 9498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:38.558079 9498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:38.558197 9498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:47:38.567481 9498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35107
--tserver_master_addrs=127.0.97.62:33411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:38.568709 9498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:38.569914 9498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:38.581995 9503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:38.582053 9506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:38.582017 9504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:38.582832 9498 server_base.cc:1034] running on GCE node
I20260929 20:47:38.583441 9498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:38.584357 9498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:38.585534 9498 hybrid_clock.cc:648] HybridClock initialized: now 1790714858585482 us; error 56 us; skew 500 ppm
I20260929 20:47:38.585808 9498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:38 dist-test-slave-blv7 krb5kdc[8629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714858, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:38.588766 9498 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 20:47:38.590130 9498 webserver.cc:460] Webserver started at http://127.0.97.3:35107/ using document root <none> and password file <none>
I20260929 20:47:38.590790 9498 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:38.590903 9498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:38.596374 9498 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 20:47:38.598984 9513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:38.600462 9498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:47:38.600608 9498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "5ba6a2566cbb4c60b6139d8ec820eceb"
format_stamp: "Formatted at 2026-09-29 20:47:36 on dist-test-slave-blv7"
server_key: "d6cc36095f12a937eb32fa21c5bcca0a"
server_key_iv: "b2ab2c8c6829f18ff506140591d69f20"
server_key_version: "encryptionkey@0"
I20260929 20:47:38.601552 9498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:38.617841 9498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:38.621565 9498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:38.621788 9498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:38.622625 9498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:38.623802 9498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:38.623881 9498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:38.623982 9498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:38.624046 9498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:38.652019 9498 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39723
I20260929 20:47:38.652081 9625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39723 every 8 connection(s)
I20260929 20:47:38.653367 9498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:47:38.660013 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 9498
Sep 29 20:47:38 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:38 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 20:47:38.668522 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:38.655704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57211 (local address 127.0.97.62:33411)
0929 20:47:38.655880 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:38.655885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:38.656912 (+ 1027us) server_negotiation.cc:407] Connection header received
0929 20:47:38.658609 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:38.658614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:38.658692 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:38.658775 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:38.660236 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.661252 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.662996 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.663485 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.663596 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:38.668236 (+ 4640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57211: BlockingRecv error: recv got EOF from 127.0.97.3:57211 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":63}
I20260929 20:47:38.669245 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:38.656117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.3:57211)
0929 20:47:38.656755 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:38.656790 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:38.658357 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:38.659010 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:38.659040 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:38.659514 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:38.660041 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:38.660057 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.661457 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.661462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:38.662863 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:38.662875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.663685 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.663692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:38.663919 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:38.664487 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:38.664508 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:38.668020 (+ 3512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":245,"threads_started":1}
W20260929 20:47:38.669945 9626 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33411 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:33411: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260929 20:47:38.674849 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:38.660888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37690 (local address 127.0.97.62:33411)
0929 20:47:38.661581 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:38.661586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:38.661601 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:47:38.661643 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:38.661647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:38.661712 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:38.661826 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:38.663200 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.663986 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.665236 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.665635 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.665736 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:38.668316 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:38.668333 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:38.668353 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:38.668395 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:38.671752 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:38.672338 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:38.672346 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:38.672354 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:38.672609 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:47:38.672940 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:47:38.672943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:47:38.672945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:47:38.673323 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:38.673530 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:38.674077 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 20:47:38.674414 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":596,"thread_start_us":494,"threads_started":1}
Sep 29 20:47:38 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:38 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:38.678114 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:38.670421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.3:57657)
0929 20:47:38.670679 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:38.670692 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:38.670762 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:38.671076 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:38.671080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:38.671436 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:38.671559 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:38.671566 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.672641 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.672645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:38.673769 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:38.673783 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.674646 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.674650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:38.674786 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:38.675268 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:38.675283 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:38.677864 (+ 2581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182}
W20260929 20:47:38.678172 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:38.670543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57657 (local address 127.0.97.62:33411)
0929 20:47:38.670712 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:38.670717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:38.670735 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:47:38.670854 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:38.670859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:38.670926 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:38.671010 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:38.671676 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.672523 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.673917 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.674520 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.674637 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:38.678013 (+ 3376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57657: BlockingRecv error: recv got EOF from 127.0.97.3:57657 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 20:47:38 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:38 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:38.685370 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:38.678751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.3:34361)
0929 20:47:38.678895 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:38.678912 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:38.678999 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:38.679286 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:38.679290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:38.679556 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:38.679685 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:38.679692 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.680646 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.680651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:38.681549 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:38.681559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.682182 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.682188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:38.682341 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:38.682749 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:38.682764 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:38.685089 (+ 2325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 20:47:38.685494 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:38.678820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34361 (local address 127.0.97.62:33411)
0929 20:47:38.679040 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:38.679045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:38.679063 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:47:38.679105 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:38.679110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:38.679179 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:38.679250 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:38.679818 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.680527 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.681670 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.682063 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.682161 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:38.685268 (+ 3107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34361: BlockingRecv error: recv got EOF from 127.0.97.3:34361 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
W20260929 20:47:38.685637 9626 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 20:47:38 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:38 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:38.846830 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:38.836646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.1:38113)
0929 20:47:38.837199 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:38.837220 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:38.837358 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:38.837688 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:38.837692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:38.838252 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:38.838503 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:38.838516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.839590 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.839596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:38.841183 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:38.841199 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.842007 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.842013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:38.842185 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:38.842791 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:38.842821 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:38.846591 (+ 3770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":292,"threads_started":1}
W20260929 20:47:38.847501 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:38.836739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38113 (local address 127.0.97.62:33411)
0929 20:47:38.836929 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:38.836935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:38.837299 (+ 364us) server_negotiation.cc:407] Connection header received
0929 20:47:38.837479 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:38.837483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:38.837545 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:38.837620 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:38.838666 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.839458 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.841352 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:38.841825 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:38.841937 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:38.847267 (+ 5330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38113: BlockingRecv error: recv got EOF from 127.0.97.1:38113 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 20:47:39 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:39 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:39.278602 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:39.270450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.2:57115)
0929 20:47:39.270991 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:39.271006 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:39.271133 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:39.271410 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:39.271413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:39.271911 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:39.272123 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:39.272137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:39.273331 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:39.273335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:39.274413 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:39.274427 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:39.275165 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:39.275170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:39.275284 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:39.275795 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:39.275817 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:39.278391 (+ 2574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":290,"threads_started":1}
W20260929 20:47:39.278851 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:39.270554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57115 (local address 127.0.97.62:33411)
0929 20:47:39.270743 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:39.270747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:39.271122 (+ 375us) server_negotiation.cc:407] Connection header received
0929 20:47:39.271220 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:39.271224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:39.271295 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:39.271387 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:39.272273 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:39.273195 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:39.274581 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:39.275008 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:39.275158 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:39.278604 (+ 3446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57115: BlockingRecv error: recv got EOF from 127.0.97.2:57115 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 20:47:39 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:39 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:39.694698 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:39.686278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.3:57427)
0929 20:47:39.686758 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:39.686775 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:39.686889 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:39.687279 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:39.687282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:39.687678 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:39.687864 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:39.687873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:39.688910 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:39.688914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:39.689855 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:39.689866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:39.690627 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:39.690632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:39.690768 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:39.691362 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:39.691384 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:39.694395 (+ 3011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":470,"threads_started":1}
W20260929 20:47:39.694844 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:39.686346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57427 (local address 127.0.97.62:33411)
0929 20:47:39.686531 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:39.686537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:39.686862 (+ 325us) server_negotiation.cc:407] Connection header received
0929 20:47:39.686995 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:39.686999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:39.687099 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:39.687240 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:39.688020 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:39.688758 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:39.689987 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:39.690445 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:39.690536 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:39.694617 (+ 4081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57427: BlockingRecv error: recv got EOF from 127.0.97.3:57427 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 20:47:39 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:39 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:39.855255 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:39.847822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.1:60531)
0929 20:47:39.848250 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:39.848267 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:39.848377 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:39.848648 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:39.848651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:39.849009 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:39.849148 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:39.849156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:39.850127 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:39.850131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:39.851101 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:39.851112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:39.851738 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:39.851742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:39.851855 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:39.852422 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:39.852438 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:39.854984 (+ 2546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":234,"threads_started":1}
W20260929 20:47:39.855434 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:39.847889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60531 (local address 127.0.97.62:33411)
0929 20:47:39.848105 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:39.848110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:39.848339 (+ 229us) server_negotiation.cc:407] Connection header received
0929 20:47:39.848464 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:39.848467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:39.848528 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:39.848607 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:39.849268 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:39.850017 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:39.851210 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:39.851629 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:39.851718 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:39.855207 (+ 3489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60531: BlockingRecv error: recv got EOF from 127.0.97.1:60531 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 29 20:47:40 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:40 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 20:47:40.288254 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:40.279659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39997 (local address 127.0.97.62:33411)
0929 20:47:40.279854 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:40.279860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:40.280112 (+ 252us) server_negotiation.cc:407] Connection header received
0929 20:47:40.280327 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:40.280332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:40.280405 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:40.280484 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:40.281259 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:40.281961 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:40.283338 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:40.283916 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:40.284043 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:40.288049 (+ 4006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39997: BlockingRecv error: recv got EOF from 127.0.97.2:39997 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:47:40.288180 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:40.279516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.2:39997)
0929 20:47:40.280022 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:40.280037 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:40.280165 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:40.280533 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:40.280539 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:40.280973 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:40.281117 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:40.281126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:40.282094 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:40.282099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:40.283210 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:40.283222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:40.284107 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:40.284115 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:40.284304 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:40.284961 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:40.284980 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:40.287854 (+ 2874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":266,"threads_started":1}
Sep 29 20:47:40 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:40 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:40.704088 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:40.695686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.3:35249)
0929 20:47:40.696161 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:40.696178 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:40.696287 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:40.696568 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:40.696572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:40.697028 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:40.697219 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:40.697230 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:40.698601 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:40.698605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:40.699659 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:40.699672 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:40.700336 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:40.700341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:40.700485 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:40.701092 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:40.701109 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:40.703764 (+ 2655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":265,"threads_started":1}
W20260929 20:47:40.704228 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:40.695816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35249 (local address 127.0.97.62:33411)
0929 20:47:40.695974 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:40.695978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:40.696238 (+ 260us) server_negotiation.cc:407] Connection header received
0929 20:47:40.696392 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:40.696396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:40.696456 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:40.696530 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:40.697392 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:40.698466 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:40.699773 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:40.700185 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:40.700279 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:40.703977 (+ 3698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35249: BlockingRecv error: recv got EOF from 127.0.97.3:35249 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 20:47:40 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:40 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:40.864514 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:40.856064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.1:60991)
0929 20:47:40.856574 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:40.856591 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:40.856746 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:40.857072 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:40.857078 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:40.857449 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:40.857614 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:40.857634 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:40.858754 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:40.858760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:40.859756 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:40.859770 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:40.860465 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:40.860471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:40.860670 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:40.861274 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:40.861293 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:40.864262 (+ 2969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":266,"threads_started":1}
W20260929 20:47:40.864706 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:40.856141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60991 (local address 127.0.97.62:33411)
0929 20:47:40.856439 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:40.856445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:40.856687 (+ 242us) server_negotiation.cc:407] Connection header received
0929 20:47:40.856870 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:40.856875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:40.856941 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:40.857027 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:40.857770 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:40.858595 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:40.859901 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:40.860325 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:40.860439 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:40.864460 (+ 4021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60991: BlockingRecv error: recv got EOF from 127.0.97.1:60991 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 29 20:47:41 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:41 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:41.297287 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:41.289236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.2:45545)
0929 20:47:41.289711 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:41.289727 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:41.289843 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:41.290159 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:41.290162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:41.290525 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:41.290682 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:41.290690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:41.291823 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:41.291827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:41.292746 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:41.292757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:41.293448 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:41.293453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:41.293585 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:41.294297 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:41.294313 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:41.297049 (+ 2736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":267,"threads_started":1}
W20260929 20:47:41.297449 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:41.289355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45545 (local address 127.0.97.62:33411)
0929 20:47:41.289516 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:41.289522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:41.289799 (+ 277us) server_negotiation.cc:407] Connection header received
0929 20:47:41.289931 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:41.289934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:41.290004 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:41.290080 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:41.290841 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:41.291696 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:41.292884 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:41.293312 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:41.293411 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:41.297254 (+ 3843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45545: BlockingRecv error: recv got EOF from 127.0.97.2:45545 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 20:47:41 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:41 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:41.712895 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:41.705015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.3:47021)
0929 20:47:41.705652 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:41.705670 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:41.705829 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:41.706217 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:41.706221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:41.706566 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:41.706734 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:41.706743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:41.707758 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:41.707764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:41.708696 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:41.708708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:41.709348 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:41.709352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:41.709488 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:41.710017 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:41.710033 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:41.712664 (+ 2631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":423,"threads_started":1}
W20260929 20:47:41.713047 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:41.705087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47021 (local address 127.0.97.62:33411)
0929 20:47:41.705247 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:41.705261 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:41.705799 (+ 538us) server_negotiation.cc:407] Connection header received
0929 20:47:41.705989 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:41.705995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:41.706092 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:41.706176 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:41.706894 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:41.707606 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:41.708828 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:41.709199 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:41.709291 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:41.712859 (+ 3568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47021: BlockingRecv error: recv got EOF from 127.0.97.3:47021 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 29 20:47:41 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:41 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:41.873728 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:41.865415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.1:59449)
0929 20:47:41.865905 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:41.865920 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:41.866037 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:41.866359 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:41.866361 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:41.866803 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:41.866997 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:41.867009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:41.868096 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:41.868100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:41.869023 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:41.869034 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:41.869897 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:41.869902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:41.870060 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:41.870625 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:41.870649 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:41.873433 (+ 2784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":265,"threads_started":1}
W20260929 20:47:41.873888 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:41.865484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59449 (local address 127.0.97.62:33411)
0929 20:47:41.865664 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:41.865668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:41.866020 (+ 352us) server_negotiation.cc:407] Connection header received
0929 20:47:41.866130 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:41.866133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:41.866228 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:41.866311 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:41.867197 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:41.867986 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:41.869177 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:41.869755 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:41.869877 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:41.873673 (+ 3796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59449: BlockingRecv error: recv got EOF from 127.0.97.1:59449 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 29 20:47:42 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:42 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:42.306790 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:42.298205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.2:59589)
0929 20:47:42.298854 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:42.298869 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:42.298985 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:42.299337 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:42.299341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:42.299709 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:42.299880 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:42.299889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:42.300915 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:42.300919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:42.301951 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:42.301963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:42.302739 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:42.302743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:42.302877 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:42.303444 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:42.303461 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:42.306462 (+ 3001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":446,"threads_started":1}
W20260929 20:47:42.306886 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:42.298340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59589 (local address 127.0.97.62:33411)
0929 20:47:42.298525 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:42.298531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:42.298949 (+ 418us) server_negotiation.cc:407] Connection header received
0929 20:47:42.299115 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:42.299120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:42.299197 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:42.299306 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:42.300007 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:42.300744 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:42.302153 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:42.302594 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:42.302689 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:42.306667 (+ 3978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59589: BlockingRecv error: recv got EOF from 127.0.97.2:59589 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 20:47:42 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:42 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:42.721735 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:42.713726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.3:34779)
0929 20:47:42.714227 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:42.714241 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:42.714361 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:42.714632 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:42.714635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:42.714999 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:42.715192 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:42.715205 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:42.716234 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:42.716238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:42.717163 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:42.717174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:42.717811 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:42.717816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:42.717962 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:42.718533 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:42.718557 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:42.721521 (+ 2964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":270,"threads_started":1}
W20260929 20:47:42.721942 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:42.713809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34779 (local address 127.0.97.62:33411)
0929 20:47:42.713963 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:42.713967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:42.714300 (+ 333us) server_negotiation.cc:407] Connection header received
0929 20:47:42.714454 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:42.714458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:42.714519 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:42.714605 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:42.715349 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:42.716106 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:42.717272 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:42.717672 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:42.717758 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:42.721702 (+ 3944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34779: BlockingRecv error: recv got EOF from 127.0.97.3:34779 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 20:47:42 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:42 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:42.883045 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:42.874759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.1:33701)
0929 20:47:42.875279 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:42.875302 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:42.875455 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:42.875855 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:42.875859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:42.876330 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:42.876504 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:42.876514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:42.877784 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:42.877790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:42.878765 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:42.878777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:42.879496 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:42.879500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:42.879635 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:42.880159 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:42.880215 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:42.882789 (+ 2574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":257,"threads_started":1}
W20260929 20:47:42.883186 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:42.874781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33701 (local address 127.0.97.62:33411)
0929 20:47:42.875044 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:42.875052 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:42.875402 (+ 350us) server_negotiation.cc:407] Connection header received
0929 20:47:42.875597 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:42.875602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:42.875666 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:42.875763 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:42.876652 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:42.877623 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:42.878902 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:42.879362 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:42.879458 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:42.882994 (+ 3536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33701: BlockingRecv error: recv got EOF from 127.0.97.1:33701 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 20:47:43 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:43 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:43.315833 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:43.307755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.2:37983)
0929 20:47:43.308389 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:43.308408 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:43.308541 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:43.308933 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:43.308937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:43.309335 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:43.309487 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:43.309495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:43.310441 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:43.310446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:43.311523 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:43.311537 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:43.312201 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:43.312206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:43.312379 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:43.312933 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:43.312950 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:43.315596 (+ 2646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":219,"threads_started":1}
W20260929 20:47:43.316020 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:43.307901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37983 (local address 127.0.97.62:33411)
0929 20:47:43.308079 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:43.308085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:43.308478 (+ 393us) server_negotiation.cc:407] Connection header received
0929 20:47:43.308740 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:43.308744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:43.308804 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:43.308877 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:43.309592 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:43.310296 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:43.311648 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:43.312066 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:43.312154 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:43.315794 (+ 3640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37983: BlockingRecv error: recv got EOF from 127.0.97.2:37983 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 20:47:43 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:43 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:43.730993 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:43.722614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.3:41513)
0929 20:47:43.723136 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:43.723152 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:43.723263 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:43.723522 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:43.723525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:43.723924 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:43.724071 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:43.724079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:43.725378 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:43.725383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:43.726560 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:43.726573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:43.727347 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:43.727352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:43.727497 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:43.728050 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:43.728065 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:43.730745 (+ 2680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":238,"threads_started":1}
W20260929 20:47:43.731108 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:43.722669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41513 (local address 127.0.97.62:33411)
0929 20:47:43.722884 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:43.722888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:43.723280 (+ 392us) server_negotiation.cc:407] Connection header received
0929 20:47:43.723344 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:43.723347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:43.723411 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:43.723497 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:43.724212 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:43.725167 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:43.726709 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:43.727134 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:43.727233 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:43.730918 (+ 3685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41513: BlockingRecv error: recv got EOF from 127.0.97.3:41513 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 29 20:47:43 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:43 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:43.891901 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:43.883974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.1:56349)
0929 20:47:43.884429 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:43.884445 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:43.884551 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:43.884876 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:43.884878 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:43.885201 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:43.885359 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:43.885367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:43.886301 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:43.886306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:43.887274 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:43.887287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:43.888138 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:43.888144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:43.888321 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:43.888895 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:43.888913 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:43.891600 (+ 2687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":247,"threads_started":1}
W20260929 20:47:43.892019 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:43.884060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56349 (local address 127.0.97.62:33411)
0929 20:47:43.884227 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:43.884231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:43.884508 (+ 277us) server_negotiation.cc:407] Connection header received
0929 20:47:43.884684 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:43.884688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:43.884759 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:43.884841 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:43.885465 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:43.886161 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:43.887425 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:43.887949 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:43.888083 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:43.891802 (+ 3719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56349: BlockingRecv error: recv got EOF from 127.0.97.1:56349 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 20:47:44 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:44 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:44.324847 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:44.316792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.2:43115)
0929 20:47:44.317294 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:44.317309 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:44.317421 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:44.317719 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:44.317723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:44.318068 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:44.318250 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:44.318259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:44.319391 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:44.319397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:44.320486 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:44.320501 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:44.321279 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:44.321284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:44.321426 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:44.322002 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:44.322020 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:44.324625 (+ 2605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":281,"threads_started":1}
W20260929 20:47:44.325075 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:44.316886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43115 (local address 127.0.97.62:33411)
0929 20:47:44.317032 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:44.317036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:44.317389 (+ 353us) server_negotiation.cc:407] Connection header received
0929 20:47:44.317523 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:44.317527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:44.317590 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:44.317677 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:44.318397 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:44.319168 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:44.320619 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:44.321137 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:44.321239 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:44.324849 (+ 3610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43115: BlockingRecv error: recv got EOF from 127.0.97.2:43115 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 20:47:44 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:44 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:44.740041 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:44.731979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.3:46783)
0929 20:47:44.732586 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:44.732602 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:44.732712 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:44.732989 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:44.732992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:44.733378 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:44.733518 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:44.733526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:44.734599 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:44.734604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:44.735745 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:44.735761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:44.736477 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:44.736483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:44.736627 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:44.737158 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:44.737176 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:44.739835 (+ 2659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":386,"threads_started":1}
W20260929 20:47:44.740234 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:44.732079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46783 (local address 127.0.97.62:33411)
0929 20:47:44.732220 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:44.732225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:44.732706 (+ 481us) server_negotiation.cc:407] Connection header received
0929 20:47:44.732810 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:44.732815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:44.732877 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:44.732962 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:44.733633 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:44.734439 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:44.735872 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:44.736329 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:44.736434 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:44.740013 (+ 3579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46783: BlockingRecv error: recv got EOF from 127.0.97.3:46783 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 20:47:44 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:44 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:44.900614 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:44.892734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.1:36037)
0929 20:47:44.893284 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:44.893300 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:44.893415 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:44.893752 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:44.893755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:44.894160 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:44.894354 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:44.894363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:44.895370 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:44.895373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:44.896258 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:44.896268 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:44.896953 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:44.896957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:44.897063 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:44.897551 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:44.897567 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:44.900322 (+ 2755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":238,"threads_started":1}
W20260929 20:47:44.900730 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:44.892830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36037 (local address 127.0.97.62:33411)
0929 20:47:44.893024 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:44.893031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:44.893397 (+ 366us) server_negotiation.cc:407] Connection header received
0929 20:47:44.893512 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:44.893515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:44.893576 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:44.893664 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:44.894481 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:44.895260 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:44.896405 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:44.896833 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:44.896928 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:44.900489 (+ 3561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36037: BlockingRecv error: recv got EOF from 127.0.97.1:36037 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 20:47:45 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:45 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:45.333894 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:45.325853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.2:33649)
0929 20:47:45.326402 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:45.326418 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:45.326530 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:45.326870 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:45.326873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:45.327367 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:45.327537 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:45.327546 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:45.328520 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:45.328523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:45.329504 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:45.329519 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:45.330367 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:45.330373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:45.330515 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:45.331062 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:45.331084 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:45.333653 (+ 2569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":228,"threads_started":1}
W20260929 20:47:45.334054 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:45.325867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33649 (local address 127.0.97.62:33411)
0929 20:47:45.326132 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:45.326139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:45.326510 (+ 371us) server_negotiation.cc:407] Connection header received
0929 20:47:45.326663 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:45.326668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:45.326742 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:45.326825 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:45.327656 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:45.328408 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:45.329660 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:45.330232 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:45.330351 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:45.333853 (+ 3502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33649: BlockingRecv error: recv got EOF from 127.0.97.2:33649 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
Sep 29 20:47:45 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:45 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:45.749729 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:45.740998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.3:42643)
0929 20:47:45.741586 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:45.741601 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:45.741719 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:45.742047 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:45.742051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:45.742407 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:45.742570 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:45.742578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:45.743710 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:45.743715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:45.744631 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:45.744644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:45.745453 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:45.745458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:45.745564 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:45.746105 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:45.746122 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:45.749488 (+ 3366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":250,"threads_started":1}
W20260929 20:47:45.749881 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:45.741105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42643 (local address 127.0.97.62:33411)
0929 20:47:45.741250 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:45.741255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:45.741660 (+ 405us) server_negotiation.cc:407] Connection header received
0929 20:47:45.741818 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:45.741822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:45.741884 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:45.741970 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:45.742690 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:45.743593 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:45.744773 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:45.745277 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:45.745384 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:45.749692 (+ 4308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42643: BlockingRecv error: recv got EOF from 127.0.97.3:42643 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 20:47:45 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:45 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:45.909543 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:45.901661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.1:46685)
0929 20:47:45.902145 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:45.902161 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:45.902288 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:45.902575 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:45.902601 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:45.903080 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:45.903229 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:45.903237 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:45.904191 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:45.904198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:45.905144 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:45.905156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:45.905874 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:45.905880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:45.906033 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:45.906549 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:45.906565 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:45.909251 (+ 2686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":252,"threads_started":1}
W20260929 20:47:45.909715 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:45.901862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46685 (local address 127.0.97.62:33411)
0929 20:47:45.902080 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:45.902086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:45.902232 (+ 146us) server_negotiation.cc:407] Connection header received
0929 20:47:45.902402 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:45.902405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:45.902463 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:45.902537 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:45.903354 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:45.904062 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:45.905278 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:45.905715 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:45.905822 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:45.909489 (+ 3667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46685: BlockingRecv error: recv got EOF from 127.0.97.1:46685 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 20:47:46 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:46 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:46.342792 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:46.334883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.2:38397)
0929 20:47:46.335368 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:46.335391 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:46.335496 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:46.335742 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:46.335744 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:46.336086 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:46.336296 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:46.336306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:46.337429 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:46.337434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:46.338396 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:46.338408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:46.339104 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:46.339109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:46.339264 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:46.339859 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:46.339877 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:46.342561 (+ 2684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":273,"threads_started":1}
W20260929 20:47:46.342969 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:46.334971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38397 (local address 127.0.97.62:33411)
0929 20:47:46.335147 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:46.335152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:46.335486 (+ 334us) server_negotiation.cc:407] Connection header received
0929 20:47:46.335574 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:46.335577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:46.335637 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:46.335724 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:46.336427 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:46.337286 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:46.338522 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:46.338941 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:46.339074 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:46.342776 (+ 3702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38397: BlockingRecv error: recv got EOF from 127.0.97.2:38397 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 20:47:46 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:46 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:46.758311 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:46.750644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.3:38199)
0929 20:47:46.751107 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:46.751122 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:46.751240 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:46.751606 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:46.751610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:46.751977 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:46.752125 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:46.752134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:46.753301 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:46.753307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:46.754406 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:46.754419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:46.755091 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:46.755095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:46.755195 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:46.755716 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:46.755732 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:46.758096 (+ 2364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":236,"threads_started":1}
W20260929 20:47:46.758490 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:46.750714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38199 (local address 127.0.97.62:33411)
0929 20:47:46.750865 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:46.750869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:46.751186 (+ 317us) server_negotiation.cc:407] Connection header received
0929 20:47:46.751380 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:46.751384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:46.751448 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:46.751539 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:46.752276 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:46.753136 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:46.754525 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:46.754944 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:46.755068 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:46.758270 (+ 3202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38199: BlockingRecv error: recv got EOF from 127.0.97.3:38199 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 20:47:46 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:46 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:46.918097 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:46.910394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.1:34559)
0929 20:47:46.910822 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:46.910837 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:46.910952 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:46.911256 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:46.911259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:46.911607 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:46.911780 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:46.911789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:46.912777 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:46.912782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:46.913741 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:46.913752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:46.914445 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:46.914450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:46.914581 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:46.915184 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:46.915202 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:46.917845 (+ 2643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":207,"threads_started":1}
W20260929 20:47:46.918229 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:46.910460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34559 (local address 127.0.97.62:33411)
0929 20:47:46.910679 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:46.910684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:46.910901 (+ 217us) server_negotiation.cc:407] Connection header received
0929 20:47:46.911070 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:46.911074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:46.911142 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:46.911220 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:46.911903 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:46.912645 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:46.913881 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:46.914320 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:46.914411 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:46.918029 (+ 3618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:34559: BlockingRecv error: recv got EOF from 127.0.97.1:34559 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 20:47:47 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:47 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:47.353071 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:47.343693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.2:55181)
0929 20:47:47.344228 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:47.344248 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:47.344380 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:47.344664 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:47.344678 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:47.345166 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:47.345362 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:47.345373 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:47.346348 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:47.346354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:47.347854 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:47.347868 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:47.348765 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:47.348771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:47.348953 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:47.349566 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:47.349586 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:47.352845 (+ 3259us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":248,"threads_started":1}
W20260929 20:47:47.353283 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:47.343818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55181 (local address 127.0.97.62:33411)
0929 20:47:47.344050 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:47.344056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:47.344332 (+ 276us) server_negotiation.cc:407] Connection header received
0929 20:47:47.344480 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:47.344483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:47.344542 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:47.344617 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:47.345504 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:47.346247 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:47.347973 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:47.348623 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:47.348711 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:47.353025 (+ 4314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55181: BlockingRecv error: recv got EOF from 127.0.97.2:55181 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 29 20:47:47 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:47 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:47.767830 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:47.759272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.3:50629)
0929 20:47:47.759806 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:47.759822 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:47.759942 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:47.760324 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:47.760327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:47.760693 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:47.760858 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:47.760867 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:47.761928 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:47.761932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:47.763115 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:47.763130 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:47.763978 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:47.763984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:47.764143 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:47.764764 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:47.764801 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:47.767593 (+ 2792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":267,"threads_started":1}
W20260929 20:47:47.767972 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:47.759337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50629 (local address 127.0.97.62:33411)
0929 20:47:47.759494 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:47.759498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:47.759892 (+ 394us) server_negotiation.cc:407] Connection header received
0929 20:47:47.760078 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:47.760082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:47.760168 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:47.760255 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:47.761025 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:47.761784 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:47.763284 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:47.763833 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:47.763932 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:47.767794 (+ 3862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50629: BlockingRecv error: recv got EOF from 127.0.97.3:50629 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 20:47:47 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:47 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:47.926898 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:47.918962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.1:51467)
0929 20:47:47.919474 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:47.919492 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:47.919614 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:47.919974 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:47.919978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:47.920310 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:47.920448 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:47.920457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:47.921463 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:47.921468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:47.922440 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:47.922474 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:47.923177 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:47.923183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:47.923329 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:47.923890 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:47.923908 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:47.926650 (+ 2742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":278,"threads_started":1}
W20260929 20:47:47.927109 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:47.919165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51467 (local address 127.0.97.62:33411)
0929 20:47:47.919408 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:47.919412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:47.919562 (+ 150us) server_negotiation.cc:407] Connection header received
0929 20:47:47.919747 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:47.919752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:47.919820 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:47.919898 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:47.920588 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:47.921320 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:47.922604 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:47.923042 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:47.923150 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:47.926857 (+ 3707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51467: BlockingRecv error: recv got EOF from 127.0.97.1:51467 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 29 20:47:48 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:48 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:48.365442 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:48.353953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.2:57817)
0929 20:47:48.354430 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:48.354445 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:48.354563 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:48.354883 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:48.354888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:48.355348 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:48.355485 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:48.355494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:48.356449 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:48.356454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:48.357372 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:48.357383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:48.358123 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:48.358129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:48.358286 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:48.362555 (+ 4269us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:48.362576 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:48.365228 (+ 2652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":269,"threads_started":1}
W20260929 20:47:48.365684 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:48.354034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57817 (local address 127.0.97.62:33411)
0929 20:47:48.354209 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:48.354215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:48.354524 (+ 309us) server_negotiation.cc:407] Connection header received
0929 20:47:48.354670 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:48.354695 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:48.354755 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:48.354832 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:48.355618 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:48.356319 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:48.357504 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:48.357988 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:48.358098 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:48.365420 (+ 7322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57817: BlockingRecv error: recv got EOF from 127.0.97.2:57817 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 20:47:48 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:48 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:48.777139 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:48.768771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.3:51667)
0929 20:47:48.769289 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:48.769305 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:48.769418 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:48.769689 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:48.769692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:48.770020 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:48.770154 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:48.770162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:48.771282 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:48.771287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:48.772560 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:48.772573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:48.773301 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:48.773309 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:48.773437 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:48.774077 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:48.774096 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:48.776836 (+ 2740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":256,"threads_started":1}
W20260929 20:47:48.777256 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:48.768931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51667 (local address 127.0.97.62:33411)
0929 20:47:48.769124 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:48.769130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:48.769374 (+ 244us) server_negotiation.cc:407] Connection header received
0929 20:47:48.769510 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:48.769514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:48.769573 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:48.769650 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:48.770314 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:48.771091 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:48.772682 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:48.773097 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:48.773182 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:48.777038 (+ 3856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51667: BlockingRecv error: recv got EOF from 127.0.97.3:51667 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 20:47:48 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:48 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:48.936056 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:48.927812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.1:43553)
0929 20:47:48.928346 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:48.928362 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:48.928468 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:48.928784 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:48.928787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:48.929160 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:48.929321 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:48.929329 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:48.930291 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:48.930296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:48.931217 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:48.931229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:48.931981 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:48.931986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:48.932123 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:48.932721 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:48.932739 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:48.935730 (+ 2991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":307,"threads_started":1}
W20260929 20:47:48.936137 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:48.927936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43553 (local address 127.0.97.62:33411)
0929 20:47:48.928148 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:48.928152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:48.928428 (+ 276us) server_negotiation.cc:407] Connection header received
0929 20:47:48.928601 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:48.928606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:48.928670 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:48.928767 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:48.929449 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:48.930171 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:48.931352 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:48.931777 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:48.931888 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:48.935939 (+ 4051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43553: BlockingRecv error: recv got EOF from 127.0.97.1:43553 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 20:47:49 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:49 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:49.374379 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:49.366331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.2:38695)
0929 20:47:49.366878 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:49.366893 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:49.367000 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:49.367324 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:49.367329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:49.367707 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:49.367864 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:49.367872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:49.369036 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:49.369041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:49.370100 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:49.370111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:49.370763 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:49.370769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:49.370931 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:49.371491 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:49.371506 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:49.374029 (+ 2523us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":313,"threads_started":1}
W20260929 20:47:49.374461 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:49.366479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38695 (local address 127.0.97.62:33411)
0929 20:47:49.366628 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:49.366632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:49.366964 (+ 332us) server_negotiation.cc:407] Connection header received
0929 20:47:49.367129 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:49.367134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:49.367200 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:49.367276 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:49.367988 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:49.368926 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:49.370217 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:49.370630 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:49.370730 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:49.374244 (+ 3514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38695: BlockingRecv error: recv got EOF from 127.0.97.2:38695 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 20:47:49 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:49 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:49.785984 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:49.778081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.3:48231)
0929 20:47:49.778552 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:49.778567 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:49.778686 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:49.779056 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:49.779061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:49.779473 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:49.779664 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:49.779673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:49.780844 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:49.780848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:49.781718 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:49.781729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:49.782387 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:49.782393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:49.782540 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:49.783114 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:49.783132 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:49.785659 (+ 2527us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":278,"threads_started":1}
W20260929 20:47:49.786042 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:49.778157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48231 (local address 127.0.97.62:33411)
0929 20:47:49.778336 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:49.778343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:49.778644 (+ 301us) server_negotiation.cc:407] Connection header received
0929 20:47:49.778797 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:49.778801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:49.778862 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:49.778943 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:49.779818 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:49.780720 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:49.781841 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:49.782251 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:49.782354 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:49.785841 (+ 3487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:48231: BlockingRecv error: recv got EOF from 127.0.97.3:48231 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 20:47:49 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:49 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:49.945734 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:49.937056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.1:48441)
0929 20:47:49.937546 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:49.937560 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:49.937693 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:49.938034 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:49.938039 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:49.938450 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:49.938634 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:49.938645 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:49.940043 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:49.940049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:49.941364 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:49.941376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:49.942072 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:49.942077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:49.942223 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:49.942728 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:49.942752 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:49.945436 (+ 2684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":269,"threads_started":1}
W20260929 20:47:49.945868 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:49.937107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48441 (local address 127.0.97.62:33411)
0929 20:47:49.937363 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:49.937369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:49.937638 (+ 269us) server_negotiation.cc:407] Connection header received
0929 20:47:49.937810 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:49.937815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:49.937902 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:49.937989 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:49.938791 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:49.939875 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:49.941492 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:49.941931 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:49.942029 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:49.945627 (+ 3598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48441: BlockingRecv error: recv got EOF from 127.0.97.1:48441 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 29 20:47:50 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:50 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:50.383337 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:50.375316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.2:37169)
0929 20:47:50.375839 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:50.375854 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:50.375960 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:50.376198 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:50.376201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:50.376531 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:50.376696 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:50.376704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:50.377682 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:50.377687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:50.378813 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:50.378828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:50.379590 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:50.379596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:50.379765 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:50.380268 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:50.380285 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:50.382995 (+ 2710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":265,"threads_started":1}
W20260929 20:47:50.383433 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:50.375318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37169 (local address 127.0.97.62:33411)
0929 20:47:50.375460 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:50.375464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:50.375911 (+ 447us) server_negotiation.cc:407] Connection header received
0929 20:47:50.376037 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:50.376041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:50.376098 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:50.376179 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:50.376818 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:50.377545 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:50.378940 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:50.379458 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:50.379565 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:50.383217 (+ 3652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37169: BlockingRecv error: recv got EOF from 127.0.97.2:37169 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 20:47:50 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:50 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:50.794600 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:50.786851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.3:36849)
0929 20:47:50.787435 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:50.787451 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:50.787560 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:50.787921 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:50.787926 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:50.788377 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:50.788580 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:50.788594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:50.789601 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:50.789605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:50.790508 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:50.790520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:50.791148 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:50.791152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:50.791286 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:50.791803 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:50.791818 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:50.794365 (+ 2547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":303,"threads_started":1}
W20260929 20:47:50.794840 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:50.786980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36849 (local address 127.0.97.62:33411)
0929 20:47:50.787151 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:50.787156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:50.787525 (+ 369us) server_negotiation.cc:407] Connection header received
0929 20:47:50.787663 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:50.787667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:50.787731 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:50.787837 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:50.788729 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:50.789469 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:50.790636 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:50.791052 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:50.791136 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:50.794567 (+ 3431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36849: BlockingRecv error: recv got EOF from 127.0.97.3:36849 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 20:47:50 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:50 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:50.954835 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:50.946715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.1:48067)
0929 20:47:50.947261 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:50.947275 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:50.947386 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:50.947679 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:50.947683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:50.948053 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:50.948202 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:50.948209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:50.949257 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:50.949262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:50.950315 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:50.950331 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:50.951281 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:50.951286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:50.951394 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:50.951940 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:50.951956 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:50.954600 (+ 2644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":267,"threads_started":1}
W20260929 20:47:50.954988 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:50.946761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48067 (local address 127.0.97.62:33411)
0929 20:47:50.946912 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:50.946916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:50.947348 (+ 432us) server_negotiation.cc:407] Connection header received
0929 20:47:50.947482 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:50.947486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:50.947560 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:50.947631 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:50.948364 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:50.949105 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:50.950458 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:50.951093 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:50.951205 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:50.954763 (+ 3558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48067: BlockingRecv error: recv got EOF from 127.0.97.1:48067 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 20:47:51 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:51 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:51.391796 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:51.384219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.2:39083)
0929 20:47:51.384697 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:51.384714 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:51.384820 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:51.385092 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:51.385094 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:51.385524 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:51.385680 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:51.385689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:51.386652 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:51.386657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:51.387605 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:51.387617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:51.388368 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:51.388373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:51.388486 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:51.389043 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:51.389060 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:51.391560 (+ 2500us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":245,"threads_started":1}
W20260929 20:47:51.391947 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:51.384437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39083 (local address 127.0.97.62:33411)
0929 20:47:51.384623 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:51.384629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:51.384786 (+ 157us) server_negotiation.cc:407] Connection header received
0929 20:47:51.384904 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:51.384908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:51.384992 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:51.385069 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:51.385810 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:51.386527 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:51.387741 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:51.388185 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:51.388344 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:51.391731 (+ 3387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39083: BlockingRecv error: recv got EOF from 127.0.97.2:39083 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 20:47:51 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:51 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:51.803786 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:51.795635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.3:34811)
0929 20:47:51.796121 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:51.796140 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:51.796266 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:51.796613 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:51.796618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:51.797092 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:51.797255 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:51.797265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:51.798253 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:51.798257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:51.799234 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:51.799246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:51.799952 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:51.799957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:51.800069 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:51.800611 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:51.800636 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:51.803531 (+ 2895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":226,"threads_started":1}
W20260929 20:47:51.803913 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:51.795692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34811 (local address 127.0.97.62:33411)
0929 20:47:51.795894 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:51.795901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:51.796219 (+ 318us) server_negotiation.cc:407] Connection header received
0929 20:47:51.796409 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:51.796413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:51.796476 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:51.796558 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:51.797394 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:51.798127 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:51.799363 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:51.799795 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:51.799901 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:51.803728 (+ 3827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34811: BlockingRecv error: recv got EOF from 127.0.97.3:34811 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 20:47:51 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:51 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:51.964084 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:51.955655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.1:53093)
0929 20:47:51.956122 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:51.956138 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:51.956329 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:51.956666 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:51.956671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:51.957042 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:51.957228 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:51.957239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:51.958451 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:51.958456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:51.959551 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:51.959565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:51.960251 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:51.960258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:51.960420 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:51.961028 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:51.961047 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:51.963739 (+ 2692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":234,"threads_started":1}
W20260929 20:47:51.964201 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:51.955742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53093 (local address 127.0.97.62:33411)
0929 20:47:51.956011 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:51.956017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:51.956213 (+ 196us) server_negotiation.cc:407] Connection header received
0929 20:47:51.956449 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:51.956455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:51.956537 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:51.956625 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:51.957372 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:51.958300 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:51.959682 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:51.960106 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:51.960221 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:51.963948 (+ 3727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53093: BlockingRecv error: recv got EOF from 127.0.97.1:53093 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Sep 29 20:47:52 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:52 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:52.400648 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:52.392569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.2:49343)
0929 20:47:52.393005 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:52.393021 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:52.393129 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:52.393407 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:52.393411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:52.393760 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:52.393932 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:52.393940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:52.395079 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:52.395083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:52.395999 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:52.396011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:52.396815 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:52.396820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:52.396959 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:52.397535 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:52.397551 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:52.400329 (+ 2778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":231,"threads_started":1}
W20260929 20:47:52.400745 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:52.392649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49343 (local address 127.0.97.62:33411)
0929 20:47:52.392801 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:52.392805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:52.393091 (+ 286us) server_negotiation.cc:407] Connection header received
0929 20:47:52.393215 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:52.393218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:52.393279 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:52.393365 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:52.394049 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:52.394916 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:52.396119 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:52.396679 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:52.396810 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:52.400533 (+ 3723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49343: BlockingRecv error: recv got EOF from 127.0.97.2:49343 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 29 20:47:52 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:52 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:52.812541 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:52.804667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.3:57299)
0929 20:47:52.805266 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:52.805285 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:52.805401 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:52.805674 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:52.805676 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:52.806059 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:52.806222 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:52.806231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:52.807352 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:52.807356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:52.808267 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:52.808278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:52.808914 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:52.808919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:52.809038 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:52.809633 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:52.809652 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:52.812297 (+ 2645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":378,"threads_started":1}
W20260929 20:47:52.812711 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:52.804715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57299 (local address 127.0.97.62:33411)
0929 20:47:52.804925 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:52.804932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:52.805386 (+ 454us) server_negotiation.cc:407] Connection header received
0929 20:47:52.805492 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:52.805496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:52.805564 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:52.805649 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:52.806416 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:52.807225 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:52.808403 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:52.808797 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:52.808885 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:52.812511 (+ 3626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57299: BlockingRecv error: recv got EOF from 127.0.97.3:57299 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 20:47:52 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:52 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:52.973444 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:52.965243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.1:42833)
0929 20:47:52.966039 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:52.966057 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:52.966185 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:52.966558 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:52.966562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:52.967039 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:52.967276 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:52.967291 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:52.968279 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:52.968283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:52.969208 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:52.969219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:52.969869 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:52.969873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:52.969973 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:52.970586 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:52.970603 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:52.973119 (+ 2516us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":560,"threads_started":1}
W20260929 20:47:52.973568 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:52.965359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42833 (local address 127.0.97.62:33411)
0929 20:47:52.965558 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:52.965564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:52.966132 (+ 568us) server_negotiation.cc:407] Connection header received
0929 20:47:52.966293 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:52.966297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:52.966414 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:52.966504 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:52.967431 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:52.968128 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:52.969331 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:52.969764 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:52.969856 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:52.973361 (+ 3505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42833: BlockingRecv error: recv got EOF from 127.0.97.1:42833 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 20:47:53 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:47:53 dist-test-slave-blv7 krb5kdc[8629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:47:53.413136 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:53.401741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33411 (local address 127.0.97.2:40117)
0929 20:47:53.402627 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:53.402643 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:53.402766 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:53.403092 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:53.403097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:53.403484 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:53.403664 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:53.403672 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:53.404638 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:53.404643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:53.405557 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:53.405569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:53.406321 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:53.406327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:53.406474 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:53.410200 (+ 3726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:47:53.410219 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:53.412894 (+ 2675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:33411: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":651,"threads_started":1}
W20260929 20:47:53.413451 9366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:53.401874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40117 (local address 127.0.97.62:33411)
0929 20:47:53.402063 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:53.402069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:53.402726 (+ 657us) server_negotiation.cc:407] Connection header received
0929 20:47:53.402881 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:53.402885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:53.402942 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:53.403042 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:53.403794 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:53.404500 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:53.405687 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:53.406172 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:53.406283 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:53.413143 (+ 6860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40117: BlockingRecv error: recv got EOF from 127.0.97.2:40117 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:47:53.661072 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 9216
I20260929 20:47:53.672865 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:53.662131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.0.97.1:37587)
0929 20:47:53.662664 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:53.662674 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:53.662705 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:47:53.662762 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:53.662768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:53.662983 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:53.663181 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:53.664619 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:53.665700 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:53.666891 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:53.667524 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:53.667637 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:53.669944 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:53.669960 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:53.669980 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:53.670017 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:53.672411 (+ 2394us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 20:47:53.672646 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57566: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":247,"threads_started":1}
W20260929 20:47:53.672953 9676 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57566: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 20:47:53.673004 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 9216: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:37587: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 20:47:53.673202 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 9216
I20260929 20:47:53.686553 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 9367
I20260929 20:47:53.698498 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:53.687551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45416 (local address 127.0.97.2:45933)
0929 20:47:53.688057 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:53.688064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:53.688094 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:47:53.688152 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:53.688158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:53.688398 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:53.688563 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:53.689878 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:53.691012 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:53.692320 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:53.692923 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:53.693012 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:53.695542 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:53.695551 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:53.695566 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:53.695601 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:53.698032 (+ 2431us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 20:47:53.698269 (+ 237us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45416: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":239,"threads_started":1}
W20260929 20:47:53.698589 9677 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45416: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 20:47:53.698781 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 9367: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:45933: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 20:47:53.698909 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 9367
I20260929 20:47:53.712586 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 9498
I20260929 20:47:53.723950 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:53.713541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.0.97.3:39723)
0929 20:47:53.714048 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:53.714056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:53.714081 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:47:53.714144 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:53.714150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:53.714366 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:53.714515 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:53.715802 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:53.716827 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:53.717958 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:53.718474 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:53.718567 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:53.720991 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:53.720999 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:53.721013 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:47:53.721050 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:53.723468 (+ 2418us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 20:47:53.723715 (+ 247us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54190: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":231,"threads_started":1}
W20260929 20:47:53.724044 9678 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54190: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 20:47:53.724104 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 9498: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:39723: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 20:47:53.724265 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 9498
I20260929 20:47:53.736737 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 9147
I20260929 20:47:53.737851 9209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:53.849105 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 9147
2026-09-29T20:47:53Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20616 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T20:47:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:47:53Z Disabled control of system clock
I20260929 20:47:53.914849 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43575
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 20:47:54.232025 9683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:54.232318 9683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:54.232409 9683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:54.239514 9683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:47:54.239617 9683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:54.239658 9683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:54.239719 9683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:47:54.239768 9683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:47:54.248733 9683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43575
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:54.249939 9683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:54.251675 9683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:54.263123 9689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:54.263185 9691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:54.263192 9688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:54.263679 9683 server_base.cc:1034] running on GCE node
I20260929 20:47:54.264473 9683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:54.265811 9683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:54.267035 9683 hybrid_clock.cc:648] HybridClock initialized: now 1790714874266963 us; error 48 us; skew 500 ppm
I20260929 20:47:54.267377 9683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:47:54.269542 9683 webserver.cc:460] Webserver started at http://127.0.97.62:41047/ using document root <none> and password file <none>
I20260929 20:47:54.270208 9683 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:54.270325 9683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:54.270632 9683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:54.273439 9683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "60e60838caa94222b3892264b432fd9a"
format_stamp: "Formatted at 2026-09-29 20:47:54 on dist-test-slave-blv7"
server_key: "4ded746ff679e52f30e07ebc73ec90e9"
server_key_iv: "fe22d53458d00aea5f6f597dd2340107"
server_key_version: "encryptionkey@0"
I20260929 20:47:54.274209 9683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "60e60838caa94222b3892264b432fd9a"
format_stamp: "Formatted at 2026-09-29 20:47:54 on dist-test-slave-blv7"
server_key: "4ded746ff679e52f30e07ebc73ec90e9"
server_key_iv: "fe22d53458d00aea5f6f597dd2340107"
server_key_version: "encryptionkey@0"
I20260929 20:47:54.279134 9683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 20:47:54.282379 9697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:54.283658 9683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:47:54.283830 9683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "60e60838caa94222b3892264b432fd9a"
format_stamp: "Formatted at 2026-09-29 20:47:54 on dist-test-slave-blv7"
server_key: "4ded746ff679e52f30e07ebc73ec90e9"
server_key_iv: "fe22d53458d00aea5f6f597dd2340107"
server_key_version: "encryptionkey@0"
I20260929 20:47:54.284013 9683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:54.297179 9683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:54.297964 9683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:54.298250 9683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:54.315788 9683 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43575
I20260929 20:47:54.315806 9748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43575 every 8 connection(s)
I20260929 20:47:54.317104 9683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:47:54.320659 9749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:54.323406 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 9683
I20260929 20:47:54.323596 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:47:54.324345 388 external_mini_cluster.cc:1428] Setting key 67c75e45dc53cf051aca549659c6bac3
I20260929 20:47:54.332672 9749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a: Bootstrap starting.
I20260929 20:47:54.335991 9749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:54.337133 9749 log.cc:824] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:54.337500 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:54.326422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46636 (local address 127.0.97.62:43575)
0929 20:47:54.327357 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:54.327368 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:54.327408 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:47:54.328965 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:54.328986 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:54.329450 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:54.329770 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:54.330360 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:54.331625 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:54.333312 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:54.333954 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:54.334093 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:54.335253 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:54.335260 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:54.335273 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:54.335768 (+ 495us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:54.335858 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:54.335879 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:54.335961 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:54.336203 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 20:47:54.336552 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":312,"threads_started":1}
I20260929 20:47:54.339828 9749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a: No bootstrap required, opened a new log
I20260929 20:47:54.346776 9749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e60838caa94222b3892264b432fd9a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43575 } }
I20260929 20:47:54.347110 9749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:54.347191 9749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e60838caa94222b3892264b432fd9a, State: Initialized, Role: FOLLOWER
I20260929 20:47:54.347690 9749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e60838caa94222b3892264b432fd9a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43575 } }
I20260929 20:47:54.347893 9749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:47:54.348003 9749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:47:54.348140 9749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:54.350097 9749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e60838caa94222b3892264b432fd9a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43575 } }
I20260929 20:47:54.350512 9749 leader_election.cc:302] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60e60838caa94222b3892264b432fd9a; no voters:
I20260929 20:47:54.351138 9749 leader_election.cc:288] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:47:54.351255 9754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:47:54.351915 9754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a [term 1 LEADER]: Becoming Leader. State: Replica: 60e60838caa94222b3892264b432fd9a, State: Running, Role: LEADER
I20260929 20:47:54.352442 9754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e60838caa94222b3892264b432fd9a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43575 } }
I20260929 20:47:54.353193 9749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:47:54.355432 9756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60e60838caa94222b3892264b432fd9a. Latest consensus state: current_term: 1 leader_uuid: "60e60838caa94222b3892264b432fd9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e60838caa94222b3892264b432fd9a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43575 } } }
I20260929 20:47:54.355721 9756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:54.355528 9755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60e60838caa94222b3892264b432fd9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e60838caa94222b3892264b432fd9a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43575 } } }
I20260929 20:47:54.355840 9755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a [sys.catalog]: This master's current role is: LEADER
W20260929 20:47:54.363447 9770 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:47:54.363560 9770 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:47:54.363677 9761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:47:54.365236 9761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:47:54.372345 9761 catalog_manager.cc:1347] Generated new cluster ID: 9f2f425d61a544e792f713797e67a6d3
I20260929 20:47:54.372454 9761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:47:54.387398 9761 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:47:54.389251 9761 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:47:54.399423 9761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60e60838caa94222b3892264b432fd9a: Generated new TSK 0
I20260929 20:47:54.400483 9761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:47:54.411985 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43575
--builtin_ntp_servers=127.0.97.20:45161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 20:47:54.734349 9773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:54.734606 9773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:54.734728 9773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:54.741734 9773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:54.741837 9773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:54.741984 9773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:47:54.750900 9773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:54.752128 9773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:54.753885 9773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:54.766424 9778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:54.766364 9779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:54.766396 9781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:54.767143 9773 server_base.cc:1034] running on GCE node
I20260929 20:47:54.767714 9773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:54.768587 9773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:54.769780 9773 hybrid_clock.cc:648] HybridClock initialized: now 1790714874769727 us; error 47 us; skew 500 ppm
I20260929 20:47:54.770077 9773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:47:54.772171 9773 webserver.cc:460] Webserver started at http://127.0.97.1:33835/ using document root <none> and password file <none>
I20260929 20:47:54.772811 9773 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:54.772907 9773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:54.773221 9773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:54.775887 9773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "44e14d4e73b642a1b5a5f5c3b040df11"
format_stamp: "Formatted at 2026-09-29 20:47:54 on dist-test-slave-blv7"
server_key: "902de2accb7e406a22e78d7569dfd908"
server_key_iv: "fa0fbdf035608b5c0a6eef4f7a65595e"
server_key_version: "encryptionkey@0"
I20260929 20:47:54.776594 9773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "44e14d4e73b642a1b5a5f5c3b040df11"
format_stamp: "Formatted at 2026-09-29 20:47:54 on dist-test-slave-blv7"
server_key: "902de2accb7e406a22e78d7569dfd908"
server_key_iv: "fa0fbdf035608b5c0a6eef4f7a65595e"
server_key_version: "encryptionkey@0"
I20260929 20:47:54.781193 9773 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 20:47:54.784389 9787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:54.785573 9773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 20:47:54.785702 9773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "44e14d4e73b642a1b5a5f5c3b040df11"
format_stamp: "Formatted at 2026-09-29 20:47:54 on dist-test-slave-blv7"
server_key: "902de2accb7e406a22e78d7569dfd908"
server_key_iv: "fa0fbdf035608b5c0a6eef4f7a65595e"
server_key_version: "encryptionkey@0"
I20260929 20:47:54.785873 9773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:54.806583 9773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:54.807416 9773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:54.807652 9773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:54.808519 9773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:54.809886 9773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:54.809975 9773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:54.810079 9773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:54.810144 9773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:54.838622 9773 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33211
I20260929 20:47:54.838730 9899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33211 every 8 connection(s)
I20260929 20:47:54.839918 9773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:47:54.844449 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 9773
I20260929 20:47:54.844604 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:47:54.845222 388 external_mini_cluster.cc:1428] Setting key ba07c886e1546a4008cda75f43f5f322
I20260929 20:47:54.848124 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43575
--builtin_ntp_servers=127.0.97.20:45161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 20:47:54.852730 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:54.842474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59285 (local address 127.0.97.62:43575)
0929 20:47:54.842919 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:54.842923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:54.843995 (+ 1072us) server_negotiation.cc:407] Connection header received
0929 20:47:54.845809 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:54.845814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:54.845883 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:54.845999 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:54.847314 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:54.848383 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:54.850059 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:54.850581 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:54.850695 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:54.851968 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:54.851975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:54.851979 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:54.852001 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:54.852044 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:54.852057 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:54.852138 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:54.852344 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 20:47:54.852437 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":211,"threads_started":1}
I20260929 20:47:54.853708 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:54.842892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43575 (local address 127.0.97.1:59285)
0929 20:47:54.843819 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:54.843861 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:54.845539 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:54.846186 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:54.846197 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:54.846312 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:54.847100 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:54.847125 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:54.848526 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:54.848530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:54.849905 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:54.849917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:54.850777 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:54.850782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:54.850984 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:54.851665 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:47:54.851668 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:54.851706 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:47:54.851783 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:54.852152 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:54.852160 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:54.852167 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:47:54.852305 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 20:47:54.852643 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":308,"threads_started":1}
I20260929 20:47:54.855315 9900 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43575
I20260929 20:47:54.855593 9900 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:54.856302 9900 heartbeater.cc:511] Master 127.0.97.62:43575 requested a full tablet report, sending...
I20260929 20:47:54.858315 9714 ts_manager.cc:194] Registered new tserver with Master: 44e14d4e73b642a1b5a5f5c3b040df11 (127.0.97.1:33211)
I20260929 20:47:54.859819 9714 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:59285
W20260929 20:47:55.155247 9904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:55.155517 9904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:55.155630 9904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:55.162775 9904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:55.162880 9904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:55.163034 9904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:47:55.172286 9904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:55.173614 9904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:55.175494 9904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:55.189064 9909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:55.189036 9912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:55.189004 9910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:55.189755 9904 server_base.cc:1034] running on GCE node
I20260929 20:47:55.190358 9904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:55.191272 9904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:55.192467 9904 hybrid_clock.cc:648] HybridClock initialized: now 1790714875192422 us; error 35 us; skew 500 ppm
I20260929 20:47:55.192755 9904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:47:55.194872 9904 webserver.cc:460] Webserver started at http://127.0.97.2:46719/ using document root <none> and password file <none>
I20260929 20:47:55.195597 9904 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:55.195677 9904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:55.195935 9904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:55.198587 9904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "cbb47bfb2bdb449194bbf829b79ddec0"
format_stamp: "Formatted at 2026-09-29 20:47:55 on dist-test-slave-blv7"
server_key: "c8752318a10e276f58b1e204c76abc5a"
server_key_iv: "3d8e1daca90f14ebc7579a0b6d83f5cf"
server_key_version: "encryptionkey@0"
I20260929 20:47:55.199342 9904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "cbb47bfb2bdb449194bbf829b79ddec0"
format_stamp: "Formatted at 2026-09-29 20:47:55 on dist-test-slave-blv7"
server_key: "c8752318a10e276f58b1e204c76abc5a"
server_key_iv: "3d8e1daca90f14ebc7579a0b6d83f5cf"
server_key_version: "encryptionkey@0"
I20260929 20:47:55.203974 9904 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:47:55.206997 9918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:55.208252 9904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 20:47:55.208395 9904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "cbb47bfb2bdb449194bbf829b79ddec0"
format_stamp: "Formatted at 2026-09-29 20:47:55 on dist-test-slave-blv7"
server_key: "c8752318a10e276f58b1e204c76abc5a"
server_key_iv: "3d8e1daca90f14ebc7579a0b6d83f5cf"
server_key_version: "encryptionkey@0"
I20260929 20:47:55.208580 9904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:55.234161 9904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:55.235282 9904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:55.235597 9904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:55.236735 9904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:55.238593 9904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:55.238708 9904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:55.238817 9904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:55.238898 9904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:55.270536 9904 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44885
I20260929 20:47:55.270684 10030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44885 every 8 connection(s)
I20260929 20:47:55.272022 9904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:47:55.279536 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 9904
I20260929 20:47:55.279690 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:47:55.280238 388 external_mini_cluster.cc:1428] Setting key e25f09328b240d45729bc82eed409670
I20260929 20:47:55.282809 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43575
--builtin_ntp_servers=127.0.97.20:45161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 20:47:55.287034 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:55.275279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57191 (local address 127.0.97.62:43575)
0929 20:47:55.275478 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:55.275484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:55.276479 (+ 995us) server_negotiation.cc:407] Connection header received
0929 20:47:55.278891 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:55.278895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:55.278962 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:55.279085 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:55.280720 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:55.281663 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:55.284023 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:55.284580 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:55.284692 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:55.286218 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:55.286224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:55.286228 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:55.286247 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:55.286281 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:55.286303 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:55.286384 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:55.286611 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 20:47:55.286698 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:47:55.288457 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:55.275208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43575 (local address 127.0.97.2:57191)
0929 20:47:55.276302 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:55.276348 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:55.278604 (+ 2256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:55.279884 (+ 1280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:55.279893 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:55.279980 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:55.280566 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:55.280583 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:55.281843 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:55.281849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:55.283831 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:55.283846 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:55.284808 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:55.284815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:55.285115 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:55.285860 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:47:55.285864 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:55.285920 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:47:55.286024 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:55.286391 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:55.286400 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:55.286408 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 20:47:55.286601 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 20:47:55.287089 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":308,"threads_started":1}
I20260929 20:47:55.289906 10031 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43575
I20260929 20:47:55.290191 10031 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:55.291083 10031 heartbeater.cc:511] Master 127.0.97.62:43575 requested a full tablet report, sending...
I20260929 20:47:55.292630 9714 ts_manager.cc:194] Registered new tserver with Master: cbb47bfb2bdb449194bbf829b79ddec0 (127.0.97.2:44885)
I20260929 20:47:55.293710 9714 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:57191
W20260929 20:47:55.602646 10034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:55.602933 10034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:55.603080 10034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:55.610153 10034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:55.610239 10034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:55.610368 10034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:47:55.619253 10034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:55.620457 10034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:55.622135 10034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:55.634171 10039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:55.634209 10042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:55.634233 10040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:55.634832 10034 server_base.cc:1034] running on GCE node
I20260929 20:47:55.635393 10034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:55.636219 10034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:55.637419 10034 hybrid_clock.cc:648] HybridClock initialized: now 1790714875637369 us; error 45 us; skew 500 ppm
I20260929 20:47:55.637714 10034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:47:55.639828 10034 webserver.cc:460] Webserver started at http://127.0.97.3:37839/ using document root <none> and password file <none>
I20260929 20:47:55.640483 10034 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:55.640575 10034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:55.640867 10034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:55.643561 10034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "c85bf18274634eb981115596aa6cc032"
format_stamp: "Formatted at 2026-09-29 20:47:55 on dist-test-slave-blv7"
server_key: "89b886e2e6b84c4ada34403e0fe99534"
server_key_iv: "7a9e820a42d0d8e909f2578af49b3bc6"
server_key_version: "encryptionkey@0"
I20260929 20:47:55.644321 10034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c85bf18274634eb981115596aa6cc032"
format_stamp: "Formatted at 2026-09-29 20:47:55 on dist-test-slave-blv7"
server_key: "89b886e2e6b84c4ada34403e0fe99534"
server_key_iv: "7a9e820a42d0d8e909f2578af49b3bc6"
server_key_version: "encryptionkey@0"
I20260929 20:47:55.649065 10034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:47:55.652386 10048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:55.653780 10034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:47:55.653946 10034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "c85bf18274634eb981115596aa6cc032"
format_stamp: "Formatted at 2026-09-29 20:47:55 on dist-test-slave-blv7"
server_key: "89b886e2e6b84c4ada34403e0fe99534"
server_key_iv: "7a9e820a42d0d8e909f2578af49b3bc6"
server_key_version: "encryptionkey@0"
I20260929 20:47:55.654124 10034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:55.680768 10034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:55.681566 10034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:55.681783 10034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:55.682643 10034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:55.684240 10034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:55.684356 10034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:55.684448 10034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:55.684509 10034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:55.713570 10034 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41141
I20260929 20:47:55.713658 10160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41141 every 8 connection(s)
I20260929 20:47:55.714777 10034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:47:55.715080 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 10034
I20260929 20:47:55.715211 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:47:55.715660 388 external_mini_cluster.cc:1428] Setting key a392acc8cc926660f01e6a1425c3bf1e
I20260929 20:47:55.726367 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:55.717242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57767 (local address 127.0.97.62:43575)
0929 20:47:55.717437 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:55.717444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:55.719123 (+ 1679us) server_negotiation.cc:407] Connection header received
0929 20:47:55.720734 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:55.720738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:55.720800 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:55.720877 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:55.721850 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:55.722545 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:55.724052 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:55.724446 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:55.724541 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:55.725768 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:55.725773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:55.725776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:55.725793 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:55.725821 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:55.725832 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:55.725897 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:55.726062 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 20:47:55.726132 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:47:55.727288 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:55.717925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43575 (local address 127.0.97.3:57767)
0929 20:47:55.718949 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:55.718999 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:55.720514 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:55.721060 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:55.721068 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:55.721157 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:55.721705 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:55.721720 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:55.722658 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:55.722661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:55.723931 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:47:55.723943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:47:55.724616 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:47:55.724621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:47:55.724819 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:55.725496 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:47:55.725499 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:55.725536 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:47:55.725612 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:55.725902 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:55.725908 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:55.725912 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 20:47:55.726036 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 20:47:55.726297 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":292,"threads_started":1}
I20260929 20:47:55.728539 10161 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43575
I20260929 20:47:55.728830 10161 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:55.729564 10161 heartbeater.cc:511] Master 127.0.97.62:43575 requested a full tablet report, sending...
I20260929 20:47:55.730841 9714 ts_manager.cc:194] Registered new tserver with Master: c85bf18274634eb981115596aa6cc032 (127.0.97.3:41141)
I20260929 20:47:55.731596 9714 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:57767
I20260929 20:47:55.742388 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 20:47:55.748019 9902 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:55.746515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46652 (local address 127.0.97.62:43575)
0929 20:47:55.746744 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:55.746750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:55.747185 (+ 435us) server_negotiation.cc:407] Connection header received
0929 20:47:55.747294 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:55.747299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:55.747368 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:55.747447 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:55.747753 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46652: BlockingRecv error: recv got EOF from 127.0.0.1:46652 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:47:55.748498 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 9773
I20260929 20:47:55.755831 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:55.749407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42508 (local address 127.0.97.1:33211)
0929 20:47:55.749897 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:55.749905 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:55.749933 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:47:55.749985 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:55.749990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:55.750189 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:55.750316 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:55.750615 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:55.751678 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:55.752864 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:55.753474 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:55.753570 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:55.754469 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:55.754477 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:55.754490 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:55.755126 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:55.755170 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:55.755189 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:55.755276 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:55.755413 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 20:47:55.755531 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":239,"threads_started":1}
I20260929 20:47:55.756996 9895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:55.836383 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 9773
I20260929 20:47:55.849639 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 9904
I20260929 20:47:55.857280 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:55.850494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42838 (local address 127.0.97.2:44885)
0929 20:47:55.851009 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:55.851036 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:55.851067 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:47:55.851125 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:55.851131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:55.851357 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:55.851488 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:55.851847 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:55.852811 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:55.854143 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:55.854744 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:55.854854 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:55.855845 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:55.855853 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:55.855866 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:55.856513 (+ 647us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:55.856559 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:55.856578 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:55.856664 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:55.856827 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 20:47:55.856941 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":238,"threads_started":1}
I20260929 20:47:55.858513 10026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:55.936548 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 9904
I20260929 20:47:55.948993 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 10034
I20260929 20:47:55.956552 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:55.949857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37704 (local address 127.0.97.3:41141)
0929 20:47:55.950431 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:55.950437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:55.950468 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:47:55.950525 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:55.950531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:55.950752 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:55.950907 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:55.951301 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:55.952324 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:55.953601 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:47:55.954158 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:47:55.954252 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:47:55.955256 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:55.955266 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:55.955280 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:55.955853 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:55.955899 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:55.955917 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:55.956003 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:55.956143 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 20:47:55.956250 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":258,"threads_started":1}
I20260929 20:47:55.957782 10156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:56.039994 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 10034
I20260929 20:47:56.052618 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 9683
I20260929 20:47:56.053799 9744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:56.151495 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 9683
2026-09-29T20:47:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2298 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T20:47:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:47:56Z Disabled control of system clock
I20260929 20:47:56.213981 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46851
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 20:47:56.520941 10180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:56.521256 10180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:56.521376 10180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:56.528390 10180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:47:56.528486 10180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:56.528539 10180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:56.528607 10180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:47:56.528666 10180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:47:56.537581 10180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46851
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:56.538812 10180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:56.540494 10180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:56.551235 10186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:56.551342 10185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:56.551317 10188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:56.551908 10180 server_base.cc:1034] running on GCE node
I20260929 20:47:56.552589 10180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:56.553886 10180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:56.555140 10180 hybrid_clock.cc:648] HybridClock initialized: now 1790714876555096 us; error 31 us; skew 500 ppm
I20260929 20:47:56.555462 10180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:47:56.557662 10180 webserver.cc:460] Webserver started at http://127.0.97.62:45029/ using document root <none> and password file <none>
I20260929 20:47:56.558341 10180 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:56.558444 10180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:56.558755 10180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:56.561643 10180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "705874e5d7044ebf96bc01f692ef1217"
format_stamp: "Formatted at 2026-09-29 20:47:56 on dist-test-slave-blv7"
server_key: "0c5538615ab2d1f356e383d28f348fd1"
server_key_iv: "2850ec0f6e2b6a16a662ac64c22bf20e"
server_key_version: "encryptionkey@0"
I20260929 20:47:56.562500 10180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "705874e5d7044ebf96bc01f692ef1217"
format_stamp: "Formatted at 2026-09-29 20:47:56 on dist-test-slave-blv7"
server_key: "0c5538615ab2d1f356e383d28f348fd1"
server_key_iv: "2850ec0f6e2b6a16a662ac64c22bf20e"
server_key_version: "encryptionkey@0"
I20260929 20:47:56.567575 10180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 20:47:56.570922 10194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:56.572297 10180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 20:47:56.572445 10180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "705874e5d7044ebf96bc01f692ef1217"
format_stamp: "Formatted at 2026-09-29 20:47:56 on dist-test-slave-blv7"
server_key: "0c5538615ab2d1f356e383d28f348fd1"
server_key_iv: "2850ec0f6e2b6a16a662ac64c22bf20e"
server_key_version: "encryptionkey@0"
I20260929 20:47:56.572661 10180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:56.578058 10180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:56.578738 10180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:56.578999 10180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:56.595362 10180 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46851
I20260929 20:47:56.595395 10245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46851 every 8 connection(s)
I20260929 20:47:56.596596 10180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:47:56.600190 10246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:47:56.602846 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 10180
I20260929 20:47:56.603055 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:47:56.603643 388 external_mini_cluster.cc:1428] Setting key 267f124b7098fbd97cc9a9f8a51ea5fb
I20260929 20:47:56.611261 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:56.605466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59686 (local address 127.0.97.62:46851)
0929 20:47:56.606355 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:56.606371 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:56.606410 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:47:56.607968 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:56.607989 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:56.608396 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:56.608704 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:56.609715 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:56.609721 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:56.609737 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:56.609766 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:56.609822 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:56.609837 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:56.609991 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:56.610130 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 20:47:56.610368 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":331,"threads_started":1}
I20260929 20:47:56.611790 10246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217: Bootstrap starting.
I20260929 20:47:56.614815 10246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217: Neither blocks nor log segments found. Creating new log.
I20260929 20:47:56.615727 10246 log.cc:824] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217: Log is configured to *not* fsync() on all Append() calls
I20260929 20:47:56.618202 10246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217: No bootstrap required, opened a new log
I20260929 20:47:56.624954 10246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705874e5d7044ebf96bc01f692ef1217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46851 } }
I20260929 20:47:56.625300 10246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:47:56.625394 10246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 705874e5d7044ebf96bc01f692ef1217, State: Initialized, Role: FOLLOWER
I20260929 20:47:56.625931 10246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705874e5d7044ebf96bc01f692ef1217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46851 } }
I20260929 20:47:56.626128 10246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:47:56.626242 10246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:47:56.626371 10246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:47:56.628383 10246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705874e5d7044ebf96bc01f692ef1217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46851 } }
I20260929 20:47:56.628842 10246 leader_election.cc:302] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 705874e5d7044ebf96bc01f692ef1217; no voters:
I20260929 20:47:56.629406 10246 leader_election.cc:288] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:47:56.629552 10251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:47:56.630237 10251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217 [term 1 LEADER]: Becoming Leader. State: Replica: 705874e5d7044ebf96bc01f692ef1217, State: Running, Role: LEADER
I20260929 20:47:56.630759 10251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705874e5d7044ebf96bc01f692ef1217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46851 } }
I20260929 20:47:56.631315 10246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:47:56.633920 10252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "705874e5d7044ebf96bc01f692ef1217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705874e5d7044ebf96bc01f692ef1217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46851 } } }
I20260929 20:47:56.633949 10253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 705874e5d7044ebf96bc01f692ef1217. Latest consensus state: current_term: 1 leader_uuid: "705874e5d7044ebf96bc01f692ef1217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705874e5d7044ebf96bc01f692ef1217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46851 } } }
I20260929 20:47:56.634418 10253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217 [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:56.634438 10252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217 [sys.catalog]: This master's current role is: LEADER
I20260929 20:47:56.635622 10259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:47:56.641000 10259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:47:56.650283 10259 catalog_manager.cc:1347] Generated new cluster ID: 5d7c0f81ef544062b0da2271473091e9
I20260929 20:47:56.650389 10259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:47:56.702447 10259 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:47:56.702864 10259 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:47:56.709029 10259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 705874e5d7044ebf96bc01f692ef1217: Generated new TSK 0
I20260929 20:47:56.709800 10259 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:47:56.725368 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46851
--builtin_ntp_servers=127.0.97.20:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 20:47:57.034199 10270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:57.034479 10270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:57.034621 10270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:57.041833 10270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:57.041931 10270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:57.042088 10270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:47:57.051196 10270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:57.052379 10270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:57.054207 10270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:57.066790 10275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:57.066895 10278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:57.066871 10276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:57.067615 10270 server_base.cc:1034] running on GCE node
I20260929 20:47:57.068172 10270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:57.069135 10270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:57.070374 10270 hybrid_clock.cc:648] HybridClock initialized: now 1790714877070321 us; error 42 us; skew 500 ppm
I20260929 20:47:57.070662 10270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:47:57.072742 10270 webserver.cc:460] Webserver started at http://127.0.97.1:39903/ using document root <none> and password file <none>
I20260929 20:47:57.073388 10270 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:57.073503 10270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:57.073777 10270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:57.076586 10270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "42a38f1e319544149ae586aa3280ac5f"
format_stamp: "Formatted at 2026-09-29 20:47:57 on dist-test-slave-blv7"
server_key: "dd7230119f62350d41883f935e52c55b"
server_key_iv: "2a00c76d9208dc11bf4ca656e46df376"
server_key_version: "encryptionkey@0"
I20260929 20:47:57.077337 10270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "42a38f1e319544149ae586aa3280ac5f"
format_stamp: "Formatted at 2026-09-29 20:47:57 on dist-test-slave-blv7"
server_key: "dd7230119f62350d41883f935e52c55b"
server_key_iv: "2a00c76d9208dc11bf4ca656e46df376"
server_key_version: "encryptionkey@0"
I20260929 20:47:57.082187 10270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260929 20:47:57.085500 10284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:57.086864 10270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:47:57.087009 10270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "42a38f1e319544149ae586aa3280ac5f"
format_stamp: "Formatted at 2026-09-29 20:47:57 on dist-test-slave-blv7"
server_key: "dd7230119f62350d41883f935e52c55b"
server_key_iv: "2a00c76d9208dc11bf4ca656e46df376"
server_key_version: "encryptionkey@0"
I20260929 20:47:57.087268 10270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:57.092454 10270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:57.093152 10270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:57.093375 10270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:57.094174 10270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:57.095455 10270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:57.095544 10270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:57.095642 10270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:57.095705 10270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:57.124818 10270 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33897
I20260929 20:47:57.124872 10396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33897 every 8 connection(s)
I20260929 20:47:57.126044 10270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:47:57.133570 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:57.128398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38359 (local address 127.0.97.62:46851)
0929 20:47:57.128777 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:57.128781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:57.129878 (+ 1097us) server_negotiation.cc:407] Connection header received
0929 20:47:57.131537 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:57.131543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:57.131607 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:57.131698 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:57.132898 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:57.132904 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:57.132909 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:57.132926 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:57.132963 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:57.132974 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:57.133070 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:57.133232 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 20:47:57.133293 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":188,"threads_started":1}
I20260929 20:47:57.134478 10400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:57.128876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46851 (local address 127.0.97.1:38359)
0929 20:47:57.129730 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:57.129773 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:57.131276 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:57.131923 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:57.131932 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:57.132021 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:57.132623 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:47:57.132625 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:57.132662 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:47:57.132743 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:57.133053 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:57.133061 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:57.133068 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:47:57.133352 (+ 284us) client_negotiation.cc:240] Negotiation successful
0929 20:47:57.133595 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":256,"threads_started":1}
I20260929 20:47:57.134829 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 10270
I20260929 20:47:57.135040 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:47:57.135741 388 external_mini_cluster.cc:1428] Setting key f7581a3bb5481f276ba215b97478ef71
I20260929 20:47:57.136104 10397 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46851
I20260929 20:47:57.136379 10397 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:57.136963 10397 heartbeater.cc:511] Master 127.0.97.62:46851 requested a full tablet report, sending...
I20260929 20:47:57.138384 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46851
--builtin_ntp_servers=127.0.97.20:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 20:47:57.138967 10211 ts_manager.cc:194] Registered new tserver with Master: 42a38f1e319544149ae586aa3280ac5f (127.0.97.1:33897)
W20260929 20:47:57.438830 10401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:57.439137 10401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:57.439270 10401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:57.446432 10401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:57.446535 10401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:57.446691 10401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:47:57.455888 10401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:57.457181 10401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:57.458920 10401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:57.471426 10409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:57.471427 10407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:57.472084 10401 server_base.cc:1034] running on GCE node
W20260929 20:47:57.472090 10406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:57.472699 10401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:57.473580 10401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:57.474779 10401 hybrid_clock.cc:648] HybridClock initialized: now 1790714877474713 us; error 58 us; skew 500 ppm
I20260929 20:47:57.475114 10401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:47:57.477265 10401 webserver.cc:460] Webserver started at http://127.0.97.2:41549/ using document root <none> and password file <none>
I20260929 20:47:57.477927 10401 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:57.478021 10401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:57.478307 10401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:57.482625 10401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "8027501eed6a4378a4947f00cd370095"
format_stamp: "Formatted at 2026-09-29 20:47:57 on dist-test-slave-blv7"
server_key: "274dbc334a89232c012e5cf67aaf9538"
server_key_iv: "4645ee8df84213e080736ec07c06e61c"
server_key_version: "encryptionkey@0"
I20260929 20:47:57.483382 10401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8027501eed6a4378a4947f00cd370095"
format_stamp: "Formatted at 2026-09-29 20:47:57 on dist-test-slave-blv7"
server_key: "274dbc334a89232c012e5cf67aaf9538"
server_key_iv: "4645ee8df84213e080736ec07c06e61c"
server_key_version: "encryptionkey@0"
I20260929 20:47:57.488363 10401 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 20:47:57.491879 10415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:57.493193 10401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:47:57.493376 10401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "8027501eed6a4378a4947f00cd370095"
format_stamp: "Formatted at 2026-09-29 20:47:57 on dist-test-slave-blv7"
server_key: "274dbc334a89232c012e5cf67aaf9538"
server_key_iv: "4645ee8df84213e080736ec07c06e61c"
server_key_version: "encryptionkey@0"
I20260929 20:47:57.493577 10401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:57.498732 10401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:57.499540 10401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:57.499769 10401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:57.500631 10401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:57.501962 10401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:57.502049 10401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:57.502152 10401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:57.502218 10401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:57.530845 10401 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33401
I20260929 20:47:57.530892 10527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33401 every 8 connection(s)
I20260929 20:47:57.532155 10401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:47:57.537874 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 10401
I20260929 20:47:57.538095 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:47:57.538558 388 external_mini_cluster.cc:1428] Setting key 0d67961960a309062b0476dc5085bf12
I20260929 20:47:57.540401 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:57.534516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58205 (local address 127.0.97.62:46851)
0929 20:47:57.534675 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:57.534679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:57.536141 (+ 1462us) server_negotiation.cc:407] Connection header received
0929 20:47:57.537958 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:57.537965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:57.538046 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:57.538158 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:57.539586 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:57.539590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:57.539593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:57.539609 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:57.539643 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:57.539656 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:57.539768 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:57.540033 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 20:47:57.540081 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:47:57.541481 10530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:57.535051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46851 (local address 127.0.97.2:58205)
0929 20:47:57.536013 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:57.536049 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:57.537657 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:57.538482 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:57.538491 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:57.538591 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:57.539310 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:47:57.539312 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:57.539350 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:47:57.539453 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:57.539748 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:57.539757 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:57.539764 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:47:57.540032 (+ 268us) client_negotiation.cc:240] Negotiation successful
0929 20:47:57.540291 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":314,"threads_started":1}
I20260929 20:47:57.541615 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46851
--builtin_ntp_servers=127.0.97.20:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 20:47:57.542740 10528 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46851
I20260929 20:47:57.543045 10528 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:57.543707 10528 heartbeater.cc:511] Master 127.0.97.62:46851 requested a full tablet report, sending...
I20260929 20:47:57.545051 10211 ts_manager.cc:194] Registered new tserver with Master: 8027501eed6a4378a4947f00cd370095 (127.0.97.2:33401)
W20260929 20:47:57.851631 10531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:57.851897 10531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:57.852006 10531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:57.859382 10531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:57.859506 10531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:57.859674 10531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:47:57.869086 10531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:57.870371 10531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:57.872175 10531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:57.886785 10536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:57.886703 10539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:57.886731 10537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:57.887339 10531 server_base.cc:1034] running on GCE node
I20260929 20:47:57.887874 10531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:57.888728 10531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:57.889956 10531 hybrid_clock.cc:648] HybridClock initialized: now 1790714877889899 us; error 31 us; skew 500 ppm
I20260929 20:47:57.890259 10531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:47:57.892463 10531 webserver.cc:460] Webserver started at http://127.0.97.3:38919/ using document root <none> and password file <none>
I20260929 20:47:57.893105 10531 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:57.893203 10531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:57.893491 10531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:57.896248 10531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "053136f31d61449abe2807ccf6278ca1"
format_stamp: "Formatted at 2026-09-29 20:47:57 on dist-test-slave-blv7"
server_key: "b9ff20dd0029e24cfef87d34d80366e6"
server_key_iv: "8d337b3a35028cb4642b1418da579ace"
server_key_version: "encryptionkey@0"
I20260929 20:47:57.896978 10531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "053136f31d61449abe2807ccf6278ca1"
format_stamp: "Formatted at 2026-09-29 20:47:57 on dist-test-slave-blv7"
server_key: "b9ff20dd0029e24cfef87d34d80366e6"
server_key_iv: "8d337b3a35028cb4642b1418da579ace"
server_key_version: "encryptionkey@0"
I20260929 20:47:57.901872 10531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:47:57.905427 10545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:57.906719 10531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:47:57.906863 10531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "053136f31d61449abe2807ccf6278ca1"
format_stamp: "Formatted at 2026-09-29 20:47:57 on dist-test-slave-blv7"
server_key: "b9ff20dd0029e24cfef87d34d80366e6"
server_key_iv: "8d337b3a35028cb4642b1418da579ace"
server_key_version: "encryptionkey@0"
I20260929 20:47:57.907089 10531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:57.913146 10531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:57.914143 10531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:57.914420 10531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:47:57.915470 10531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:47:57.917121 10531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:47:57.917208 10531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:57.917328 10531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:47:57.917400 10531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:57.946718 10531 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33741
I20260929 20:47:57.946772 10657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33741 every 8 connection(s)
I20260929 20:47:57.948046 10531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:47:57.951984 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 10531
I20260929 20:47:57.952152 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:47:57.952562 388 external_mini_cluster.cc:1428] Setting key 93d50af72a03c866d4d2571ef2294ccc
I20260929 20:47:57.955807 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:57.950412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52303 (local address 127.0.97.62:46851)
0929 20:47:57.950597 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:57.950602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:57.951969 (+ 1367us) server_negotiation.cc:407] Connection header received
0929 20:47:57.953682 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:57.953687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:57.953769 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:57.953853 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:57.955137 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:57.955144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:57.955148 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:57.955166 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:57.955197 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:57.955208 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:57.955308 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:57.955475 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 20:47:57.955534 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:47:57.956691 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:57.950925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46851 (local address 127.0.97.3:52303)
0929 20:47:57.951821 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0929 20:47:57.951863 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:47:57.953430 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:47:57.954082 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:47:57.954091 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:47:57.954182 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:47:57.954818 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:47:57.954821 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:47:57.954871 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:47:57.954973 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:47:57.955293 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:47:57.955303 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:47:57.955310 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:47:57.955548 (+ 238us) client_negotiation.cc:240] Negotiation successful
0929 20:47:57.955792 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":260,"threads_started":1}
I20260929 20:47:57.957798 10658 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46851
I20260929 20:47:57.958060 10658 heartbeater.cc:464] Registering TS with master...
I20260929 20:47:57.958585 10658 heartbeater.cc:511] Master 127.0.97.62:46851 requested a full tablet report, sending...
I20260929 20:47:57.959842 10211 ts_manager.cc:194] Registered new tserver with Master: 053136f31d61449abe2807ccf6278ca1 (127.0.97.3:33741)
I20260929 20:47:57.968505 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 20:47:57.974259 10399 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:47:57.972865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.0.97.62:46851)
0929 20:47:57.973046 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:57.973053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:57.973509 (+ 456us) server_negotiation.cc:407] Connection header received
0929 20:47:57.973637 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:57.973641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:57.973692 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:57.973783 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:57.974030 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51868: BlockingRecv error: recv got EOF from 127.0.0.1:51868 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:47:57.974665 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 10270
I20260929 20:47:57.977798 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:57.975626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57858 (local address 127.0.97.1:33897)
0929 20:47:57.976101 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:57.976107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:57.976131 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:47:57.976184 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:57.976190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:57.976419 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:57.976529 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:57.977251 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:57.977259 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:57.977270 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:57.977285 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:57.977321 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:57.977334 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:57.977429 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:57.977504 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 20:47:57.977575 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":211,"threads_started":1}
I20260929 20:47:57.979053 10392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:58.056977 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 10270
I20260929 20:47:58.070446 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 10401
I20260929 20:47:58.073825 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:58.071285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57902 (local address 127.0.97.2:33401)
0929 20:47:58.071778 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:58.071785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:58.071812 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:47:58.071873 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:58.071879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:58.072094 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:58.072229 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:58.073153 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:58.073159 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:58.073171 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:58.073187 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:58.073226 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:58.073239 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:58.073362 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:58.073483 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 20:47:58.073581 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":241,"threads_started":1}
I20260929 20:47:58.075155 10523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:58.153755 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 10401
I20260929 20:47:58.166420 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 10531
I20260929 20:47:58.169978 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:47:58.167409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47644 (local address 127.0.97.3:33741)
0929 20:47:58.168038 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0929 20:47:58.168045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:47:58.168074 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:47:58.168131 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:47:58.168138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:47:58.168405 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:47:58.168528 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:47:58.169393 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:47:58.169399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:47:58.169410 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:47:58.169426 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:47:58.169463 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:47:58.169476 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:47:58.169600 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 20:47:58.169648 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 20:47:58.169722 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":330,"threads_started":1}
I20260929 20:47:58.171200 10653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:58.251081 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 10531
I20260929 20:47:58.263578 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 10180
I20260929 20:47:58.264781 10241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:47:58.361675 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 10180
2026-09-29T20:47:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2209 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:47:58 dist-test-slave-blv7 krb5kdc[10676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:47:58 dist-test-slave-blv7 krb5kdc[10676](info): set up 2 sockets
Sep 29 20:47:58 dist-test-slave-blv7 krb5kdc[10676](info): commencing operation
krb5kdc: starting...
W20260929 20:47:59.371647 388 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:47:59 dist-test-slave-blv7 krb5kdc[10676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:47:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:47:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:47:59.582871 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43461
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:47:59.896754 10689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:47:59.897028 10689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:47:59.897151 10689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:47:59.904279 10689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:47:59.904386 10689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:47:59.904455 10689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:47:59.904523 10689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:47:59.904564 10689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:47:59.914629 10689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:47:59.915975 10689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:47:59.917232 10689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:47:59.928602 10697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:59.928562 10694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:47:59.928625 10695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:47:59.929257 10689 server_base.cc:1034] running on GCE node
I20260929 20:47:59.930007 10689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:47:59.931406 10689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:47:59.932618 10689 hybrid_clock.cc:648] HybridClock initialized: now 1790714879932562 us; error 39 us; skew 500 ppm
I20260929 20:47:59.933009 10689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:47:59 dist-test-slave-blv7 krb5kdc[10676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:47:59.936057 10689 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:47:59.937578 10689 webserver.cc:460] Webserver started at http://127.0.97.62:36265/ using document root <none> and password file <none>
I20260929 20:47:59.938279 10689 fs_manager.cc:359] Metadata directory not provided
I20260929 20:47:59.938409 10689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:47:59.938711 10689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:47:59.941880 10689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "e5443aa91e484a6f95403b7ed5eeea5d"
format_stamp: "Formatted at 2026-09-29 20:47:59 on dist-test-slave-blv7"
server_key: "fcd05dc9792ee048561e4a1e3e655a5d"
server_key_iv: "61f5154a8b8257d1bbabb61dd028b862"
server_key_version: "encryptionkey@0"
I20260929 20:47:59.942667 10689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "e5443aa91e484a6f95403b7ed5eeea5d"
format_stamp: "Formatted at 2026-09-29 20:47:59 on dist-test-slave-blv7"
server_key: "fcd05dc9792ee048561e4a1e3e655a5d"
server_key_iv: "61f5154a8b8257d1bbabb61dd028b862"
server_key_version: "encryptionkey@0"
I20260929 20:47:59.947994 10689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 20:47:59.951565 10704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:47:59.952905 10689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:47:59.953064 10689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "e5443aa91e484a6f95403b7ed5eeea5d"
format_stamp: "Formatted at 2026-09-29 20:47:59 on dist-test-slave-blv7"
server_key: "fcd05dc9792ee048561e4a1e3e655a5d"
server_key_iv: "61f5154a8b8257d1bbabb61dd028b862"
server_key_version: "encryptionkey@0"
I20260929 20:47:59.953239 10689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:47:59.981241 10689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:47:59.985138 10689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:47:59.985387 10689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:48:00.002575 10689 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43461
I20260929 20:48:00.002605 10755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43461 every 8 connection(s)
I20260929 20:48:00.003923 10689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:48:00.007699 10756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:48:00.012534 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 10689
I20260929 20:48:00.012751 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:48:00.013509 388 external_mini_cluster.cc:1428] Setting key d6fa77e35304ca627c346034144f7077
I20260929 20:48:00.019290 10756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d: Bootstrap starting.
I20260929 20:48:00.022404 10756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d: Neither blocks nor log segments found. Creating new log.
I20260929 20:48:00.023484 10756 log.cc:824] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:48:00.026391 10756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d: No bootstrap required, opened a new log
Sep 29 20:48:00 dist-test-slave-blv7 krb5kdc[10676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:48:00.036886 10759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:00.015728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54384 (local address 127.0.97.62:43461)
0929 20:48:00.016641 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:00.016653 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:00.016693 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:48:00.018354 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:00.018378 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:00.018776 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:00.019183 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:48:00.020567 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:00.021866 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:00.023185 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:00.023932 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:00.024131 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:00.028054 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:48:00.028068 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:48:00.028094 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:48:00.028146 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:48:00.032130 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:00.032893 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:00.032902 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:00.032911 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:00.033124 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:00.033460 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:00.033465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:00.033468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:00.034200 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:48:00.034493 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:00.034913 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 20:48:00.035433 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":381,"threads_started":1}
I20260929 20:48:00.036615 10756 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5443aa91e484a6f95403b7ed5eeea5d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43461 } }
I20260929 20:48:00.037065 10756 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:48:00.037169 10756 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5443aa91e484a6f95403b7ed5eeea5d, State: Initialized, Role: FOLLOWER
I20260929 20:48:00.037887 10756 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5443aa91e484a6f95403b7ed5eeea5d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43461 } }
I20260929 20:48:00.038149 10756 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:48:00.038276 10756 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:48:00.038458 10756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:48:00.040933 10756 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5443aa91e484a6f95403b7ed5eeea5d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43461 } }
I20260929 20:48:00.041443 10756 leader_election.cc:302] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5443aa91e484a6f95403b7ed5eeea5d; no voters:
I20260929 20:48:00.042002 10756 leader_election.cc:288] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:48:00.042129 10761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:48:00.042770 10761 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d [term 1 LEADER]: Becoming Leader. State: Replica: e5443aa91e484a6f95403b7ed5eeea5d, State: Running, Role: LEADER
I20260929 20:48:00.043370 10761 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5443aa91e484a6f95403b7ed5eeea5d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43461 } }
I20260929 20:48:00.044196 10756 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:48:00.046181 10762 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5443aa91e484a6f95403b7ed5eeea5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5443aa91e484a6f95403b7ed5eeea5d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43461 } } }
I20260929 20:48:00.046216 10763 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5443aa91e484a6f95403b7ed5eeea5d. Latest consensus state: current_term: 1 leader_uuid: "e5443aa91e484a6f95403b7ed5eeea5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5443aa91e484a6f95403b7ed5eeea5d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43461 } } }
I20260929 20:48:00.046594 10762 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d [sys.catalog]: This master's current role is: LEADER
I20260929 20:48:00.046607 10763 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d [sys.catalog]: This master's current role is: LEADER
I20260929 20:48:00.047435 10770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:48:00.053483 10770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:48:00.062049 10770 catalog_manager.cc:1347] Generated new cluster ID: 1435441ae6824410ad3f506ed21d4e49
I20260929 20:48:00.062177 10770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:48:00.072517 10770 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:48:00.073804 10770 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:48:00.085829 10770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d: Generated new TSK 0
I20260929 20:48:00.086691 10770 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:48:00.147121 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43461
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:48:00.467239 10782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:48:00.467528 10782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:48:00.467648 10782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:48:00.475312 10782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:48:00.475428 10782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:48:00.475592 10782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:48:00.485348 10782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:48:00.486639 10782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:48:00.488027 10782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:48:00.500803 10790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:48:00.500910 10788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:48:00.500931 10787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:48:00.501592 10782 server_base.cc:1034] running on GCE node
I20260929 20:48:00.502189 10782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:48:00.503229 10782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:48:00.504441 10782 hybrid_clock.cc:648] HybridClock initialized: now 1790714880504361 us; error 69 us; skew 500 ppm
I20260929 20:48:00.504743 10782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:48:00 dist-test-slave-blv7 krb5kdc[10676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:48:00.507771 10782 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:48:00.509164 10782 webserver.cc:460] Webserver started at http://127.0.97.1:40807/ using document root <none> and password file <none>
I20260929 20:48:00.509861 10782 fs_manager.cc:359] Metadata directory not provided
I20260929 20:48:00.509970 10782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:48:00.510234 10782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:48:00.513078 10782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "0e0d0ce1e5c54fa9b7a816955a035feb"
format_stamp: "Formatted at 2026-09-29 20:48:00 on dist-test-slave-blv7"
server_key: "13a9d612578255aab53f00b669cf0e65"
server_key_iv: "9f358c62c0fb9429e6d70607e80a54db"
server_key_version: "encryptionkey@0"
I20260929 20:48:00.513787 10782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0e0d0ce1e5c54fa9b7a816955a035feb"
format_stamp: "Formatted at 2026-09-29 20:48:00 on dist-test-slave-blv7"
server_key: "13a9d612578255aab53f00b669cf0e65"
server_key_iv: "9f358c62c0fb9429e6d70607e80a54db"
server_key_version: "encryptionkey@0"
I20260929 20:48:00.518716 10782 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 20:48:00.522145 10797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:48:00.523568 10782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 20:48:00.523721 10782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "0e0d0ce1e5c54fa9b7a816955a035feb"
format_stamp: "Formatted at 2026-09-29 20:48:00 on dist-test-slave-blv7"
server_key: "13a9d612578255aab53f00b669cf0e65"
server_key_iv: "9f358c62c0fb9429e6d70607e80a54db"
server_key_version: "encryptionkey@0"
I20260929 20:48:00.523898 10782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:48:00.546108 10782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:48:00.550221 10782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:48:00.550478 10782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:48:00.551429 10782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:48:00.552716 10782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:48:00.552793 10782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:48:00.552908 10782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:48:00.552959 10782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:48:00.581871 10782 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32779
I20260929 20:48:00.581934 10909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32779 every 8 connection(s)
I20260929 20:48:00.583082 10782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:48:00.588342 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 10782
I20260929 20:48:00.588549 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:48:00.589282 388 external_mini_cluster.cc:1428] Setting key 3983fc387da87f809f152a9c43e5244f
Sep 29 20:48:00 dist-test-slave-blv7 krb5kdc[10676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:48:00.603832 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:00.585557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56933 (local address 127.0.97.62:43461)
0929 20:48:00.585941 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:00.585946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:00.587117 (+ 1171us) server_negotiation.cc:407] Connection header received
0929 20:48:00.589500 (+ 2383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:00.589505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:00.589576 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:00.589674 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:48:00.591412 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:00.592385 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:00.593880 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:00.594372 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:00.594482 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:00.597714 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:48:00.597719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:48:00.597722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:48:00.597760 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:48:00.600683 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:00.601537 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:00.601541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:00.601545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:00.601674 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:00.602042 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:00.602045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:00.602048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:00.602354 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:48:00.602487 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:00.603264 (+ 777us) server_negotiation.cc:299] Negotiation successful
0929 20:48:00.603472 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":186,"threads_started":1}
I20260929 20:48:00.604791 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:00.586104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43461 (local address 127.0.97.1:56933)
0929 20:48:00.586936 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:00.586977 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:48:00.589129 (+ 2152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:48:00.589947 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:48:00.589956 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:48:00.590574 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:48:00.591239 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:00.591267 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:00.592541 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:00.592546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:00.593745 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:00.593758 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:00.594583 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:00.594588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:00.594795 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:00.595477 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:48:00.595499 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:48:00.597509 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:48:00.600895 (+ 3386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:48:00.600902 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:48:00.600917 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:48:00.601414 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:48:00.601764 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:48:00.601768 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:48:00.601770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:48:00.601945 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:48:00.602501 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:48:00.602508 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:48:00.602840 (+ 332us) client_negotiation.cc:769] Sending connection context
0929 20:48:00.603156 (+ 316us) client_negotiation.cc:240] Negotiation successful
0929 20:48:00.603622 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":309,"threads_started":1}
I20260929 20:48:00.606237 10910 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43461
I20260929 20:48:00.606552 10910 heartbeater.cc:464] Registering TS with master...
I20260929 20:48:00.607414 10910 heartbeater.cc:511] Master 127.0.97.62:43461 requested a full tablet report, sending...
I20260929 20:48:00.609582 10721 ts_manager.cc:194] Registered new tserver with Master: 0e0d0ce1e5c54fa9b7a816955a035feb (127.0.97.1:32779)
I20260929 20:48:00.611037 10721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56933
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:48:00.640332 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43461
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:48:00.952488 10916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:48:00.952767 10916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:48:00.952884 10916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:48:00.960213 10916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:48:00.960317 10916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:48:00.960464 10916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:48:00.970013 10916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:48:00.971268 10916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:48:00.972528 10916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:48:00.986064 10922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:48:00.986095 10921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:48:00.986122 10924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:48:00.988540 10916 server_base.cc:1034] running on GCE node
I20260929 20:48:00.989146 10916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:48:00.990094 10916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:48:00.991291 10916 hybrid_clock.cc:648] HybridClock initialized: now 1790714880991197 us; error 89 us; skew 500 ppm
I20260929 20:48:00.991590 10916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:48:00 dist-test-slave-blv7 krb5kdc[10676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:48:00.994652 10916 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:48:00.996184 10916 webserver.cc:460] Webserver started at http://127.0.97.2:34045/ using document root <none> and password file <none>
I20260929 20:48:00.996888 10916 fs_manager.cc:359] Metadata directory not provided
I20260929 20:48:00.996989 10916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:48:00.997264 10916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:48:01.000396 10916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "5056dc9e784748499153f5843aeff2f9"
format_stamp: "Formatted at 2026-09-29 20:48:00 on dist-test-slave-blv7"
server_key: "1d6edd573bbe6b0fbf715cda59e55f52"
server_key_iv: "802759bddde852d75b10741809d1029a"
server_key_version: "encryptionkey@0"
I20260929 20:48:01.001132 10916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5056dc9e784748499153f5843aeff2f9"
format_stamp: "Formatted at 2026-09-29 20:48:00 on dist-test-slave-blv7"
server_key: "1d6edd573bbe6b0fbf715cda59e55f52"
server_key_iv: "802759bddde852d75b10741809d1029a"
server_key_version: "encryptionkey@0"
I20260929 20:48:01.006354 10916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:48:01.010164 10931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:48:01.011740 10916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:48:01.011896 10916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "5056dc9e784748499153f5843aeff2f9"
format_stamp: "Formatted at 2026-09-29 20:48:00 on dist-test-slave-blv7"
server_key: "1d6edd573bbe6b0fbf715cda59e55f52"
server_key_iv: "802759bddde852d75b10741809d1029a"
server_key_version: "encryptionkey@0"
I20260929 20:48:01.012073 10916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:48:01.037825 10916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:48:01.042227 10916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:48:01.042498 10916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:48:01.043498 10916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:48:01.044863 10916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:48:01.044939 10916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:48:01.045053 10916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:48:01.045107 10916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:48:01.075335 10916 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34903
I20260929 20:48:01.075387 11043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34903 every 8 connection(s)
I20260929 20:48:01.076555 10916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:48:01.080801 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 10916
I20260929 20:48:01.081005 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:48:01.081501 388 external_mini_cluster.cc:1428] Setting key 3744f77d11944125955b76f073cf7578
Sep 29 20:48:01 dist-test-slave-blv7 krb5kdc[10676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:48:01.096645 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:01.079003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57581 (local address 127.0.97.62:43461)
0929 20:48:01.079223 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:01.079229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:01.080309 (+ 1080us) server_negotiation.cc:407] Connection header received
0929 20:48:01.082586 (+ 2277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:01.082591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:01.082660 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:01.082747 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:48:01.084602 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.085306 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.087297 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.087676 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.087771 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:01.091554 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:48:01.091559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:48:01.091563 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:48:01.091592 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:48:01.093852 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:01.094516 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:01.094520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:01.094523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:01.094640 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:01.095057 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:01.095062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:01.095066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:01.095306 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:48:01.095432 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:01.096091 (+ 659us) server_negotiation.cc:299] Negotiation successful
0929 20:48:01.096316 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 20:48:01.097621 11046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:01.079489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43461 (local address 127.0.97.2:57581)
0929 20:48:01.080144 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:01.080185 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:48:01.082338 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:48:01.083004 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:48:01.083034 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:48:01.083641 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:48:01.084420 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:01.084446 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.085447 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.085452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:01.087177 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:01.087190 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.087877 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.087882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:01.088131 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:01.088794 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:48:01.088820 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:48:01.091340 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:48:01.093973 (+ 2633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:48:01.093982 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:48:01.094001 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:48:01.094402 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:48:01.094744 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:48:01.094749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:48:01.094753 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:48:01.094928 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:48:01.095418 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:48:01.095426 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:48:01.095753 (+ 327us) client_negotiation.cc:769] Sending connection context
0929 20:48:01.095985 (+ 232us) client_negotiation.cc:240] Negotiation successful
0929 20:48:01.096525 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":259,"threads_started":1}
I20260929 20:48:01.099181 11044 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43461
I20260929 20:48:01.099527 11044 heartbeater.cc:464] Registering TS with master...
I20260929 20:48:01.100206 11044 heartbeater.cc:511] Master 127.0.97.62:43461 requested a full tablet report, sending...
I20260929 20:48:01.101743 10721 ts_manager.cc:194] Registered new tserver with Master: 5056dc9e784748499153f5843aeff2f9 (127.0.97.2:34903)
I20260929 20:48:01.102852 10721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57581
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:48:01.130453 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43461
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:48:01.445713 11049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:48:01.445991 11049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:48:01.446082 11049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:48:01.453333 11049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:48:01.453457 11049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:48:01.453575 11049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:48:01.463449 11049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:48:01.464725 11049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:48:01.465984 11049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:48:01.479586 11057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:48:01.479614 11054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:48:01.479555 11055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:48:01.481159 11049 server_base.cc:1034] running on GCE node
I20260929 20:48:01.481756 11049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:48:01.482697 11049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:48:01.483914 11049 hybrid_clock.cc:648] HybridClock initialized: now 1790714881483866 us; error 39 us; skew 500 ppm
I20260929 20:48:01.484205 11049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:48:01 dist-test-slave-blv7 krb5kdc[10676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:48:01.487221 11049 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:48:01.488647 11049 webserver.cc:460] Webserver started at http://127.0.97.3:33991/ using document root <none> and password file <none>
I20260929 20:48:01.489341 11049 fs_manager.cc:359] Metadata directory not provided
I20260929 20:48:01.489457 11049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:48:01.489739 11049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:48:01.492715 11049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "fa40516ea7ae46f68f0dea8571e933e2"
format_stamp: "Formatted at 2026-09-29 20:48:01 on dist-test-slave-blv7"
server_key: "b8c3061027809229092d6a973c5f5da9"
server_key_iv: "c6eb6ec256bf2dc39ed904e46cd15006"
server_key_version: "encryptionkey@0"
I20260929 20:48:01.493448 11049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fa40516ea7ae46f68f0dea8571e933e2"
format_stamp: "Formatted at 2026-09-29 20:48:01 on dist-test-slave-blv7"
server_key: "b8c3061027809229092d6a973c5f5da9"
server_key_iv: "c6eb6ec256bf2dc39ed904e46cd15006"
server_key_version: "encryptionkey@0"
I20260929 20:48:01.498736 11049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:48:01.502487 11064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:48:01.504009 11049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:48:01.504190 11049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "fa40516ea7ae46f68f0dea8571e933e2"
format_stamp: "Formatted at 2026-09-29 20:48:01 on dist-test-slave-blv7"
server_key: "b8c3061027809229092d6a973c5f5da9"
server_key_iv: "c6eb6ec256bf2dc39ed904e46cd15006"
server_key_version: "encryptionkey@0"
I20260929 20:48:01.504420 11049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:48:01.520002 11049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:48:01.524231 11049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:48:01.524530 11049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:48:01.525506 11049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:48:01.526918 11049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:48:01.527006 11049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:48:01.527168 11049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:48:01.527225 11049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:48:01.558449 11049 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37621
I20260929 20:48:01.558495 11176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37621 every 8 connection(s)
I20260929 20:48:01.559795 11049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:48:01.560906 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 11049
I20260929 20:48:01.561111 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:48:01.561646 388 external_mini_cluster.cc:1428] Setting key 92e92c3a0daab803230740bd16757783
Sep 29 20:48:01 dist-test-slave-blv7 krb5kdc[10676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:48:01.578833 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:01.562674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38581 (local address 127.0.97.62:43461)
0929 20:48:01.562889 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:01.562896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:01.564016 (+ 1120us) server_negotiation.cc:407] Connection header received
0929 20:48:01.565839 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:01.565844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:01.565925 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:01.566010 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:48:01.567463 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.568188 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.569645 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.570048 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.570147 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:01.573352 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:48:01.573356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:48:01.573360 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:48:01.573396 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:48:01.576112 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:01.576883 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:01.576888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:01.576892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:01.577027 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:01.577437 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:01.577440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:01.577442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:01.577698 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:48:01.577824 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:01.578427 (+ 603us) server_negotiation.cc:299] Negotiation successful
0929 20:48:01.578578 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 20:48:01.579784 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:01.563152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43461 (local address 127.0.97.3:38581)
0929 20:48:01.563879 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:01.563916 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:48:01.565591 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:48:01.566240 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:48:01.566248 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:48:01.566733 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:48:01.567307 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:01.567333 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.568331 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.568335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:01.569519 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:01.569530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.570244 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.570251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:01.570476 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:01.571108 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:48:01.571130 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:48:01.573160 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:48:01.576252 (+ 3092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:48:01.576260 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:48:01.576276 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:48:01.576769 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:48:01.577150 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:48:01.577154 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:48:01.577156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:48:01.577341 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:48:01.577810 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:48:01.577817 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:48:01.578113 (+ 296us) client_negotiation.cc:769] Sending connection context
0929 20:48:01.578373 (+ 260us) client_negotiation.cc:240] Negotiation successful
0929 20:48:01.578750 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":301,"threads_started":1}
I20260929 20:48:01.581110 11177 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43461
I20260929 20:48:01.581415 11177 heartbeater.cc:464] Registering TS with master...
I20260929 20:48:01.582135 11177 heartbeater.cc:511] Master 127.0.97.62:43461 requested a full tablet report, sending...
I20260929 20:48:01.583434 10721 ts_manager.cc:194] Registered new tserver with Master: fa40516ea7ae46f68f0dea8571e933e2 (127.0.97.3:37621)
I20260929 20:48:01.584196 10721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38581
I20260929 20:48:01.589182 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:48:01.604728 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:01.593211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54390 (local address 127.0.97.62:43461)
0929 20:48:01.593410 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:01.593414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:01.593782 (+ 368us) server_negotiation.cc:407] Connection header received
0929 20:48:01.593925 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:01.593929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:01.593986 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:01.594055 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:48:01.595311 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.595991 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.597106 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.597550 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.597664 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:01.599857 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:48:01.599862 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:48:01.599866 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:48:01.599896 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:48:01.602391 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:01.603082 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:01.603086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:01.603088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:01.603208 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:01.603551 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:01.603554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:01.603556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:01.603871 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:48:01.604040 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:01.604361 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 20:48:01.604513 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:48:01.614135 10910 heartbeater.cc:503] Master 127.0.97.62:43461 was elected leader, sending a full tablet report...
I20260929 20:48:01.608511 10721 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:48:01.615926 10721 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:48:01.639510 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:01.627566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37621 (local address 127.0.0.1:55492)
0929 20:48:01.629221 (+ 1655us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:01.629238 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:48:01.629330 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:48:01.630619 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:48:01.630625 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:48:01.630654 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:48:01.631101 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:01.631114 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.633263 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.633269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:01.636011 (+ 2742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:01.636028 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.638364 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.638371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:01.638578 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:01.638634 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 20:48:01.638771 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 20:48:01.638992 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1530,"mutex_wait_us":29,"spinlock_wait_cycles":9600,"thread_start_us":908,"threads_started":1}
I20260929 20:48:01.640538 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:01.628214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55492 (local address 127.0.97.3:37621)
0929 20:48:01.629871 (+ 1657us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:01.629879 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:01.629919 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:48:01.629994 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:01.629999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:01.630205 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:01.630376 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:48:01.631424 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.633098 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.636225 (+ 3127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.637820 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.637951 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:01.638207 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:01.639981 (+ 1774us) server_negotiation.cc:299] Negotiation successful
0929 20:48:01.640187 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1499,"thread_start_us":1357,"threads_started":1}
I20260929 20:48:01.642411 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:01.627552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32779 (local address 127.0.0.1:33064)
0929 20:48:01.628421 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:01.628464 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:48:01.628652 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:48:01.630391 (+ 1739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:48:01.630398 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:48:01.630430 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:48:01.632031 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:01.632044 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.634094 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.635192 (+ 1098us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:01.637539 (+ 2347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:01.637556 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.641526 (+ 3970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.641533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:01.641731 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:01.641767 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 20:48:01.641854 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 20:48:01.641999 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":1,"thread_start_us":1373,"threads_started":1}
I20260929 20:48:01.645053 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:01.627761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33064 (local address 127.0.97.1:32779)
0929 20:48:01.629573 (+ 1812us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:01.629582 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:01.629611 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:48:01.629706 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:01.629714 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:01.629933 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:01.630113 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:48:01.632265 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.633919 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.637743 (+ 3824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.640976 (+ 3233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.641130 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:01.641378 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:01.644488 (+ 3110us) server_negotiation.cc:299] Negotiation successful
0929 20:48:01.644695 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1637,"thread_start_us":1478,"threads_started":1}
I20260929 20:48:01.644968 11112 tablet_service.cc:1467] Processing CreateTablet for tablet abfca6c37ff2458391cb34610e6147f8 (DEFAULT_TABLE table=test-table [id=b09e720fe7c54782a1ad2f86744afccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:48:01.647425 11112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abfca6c37ff2458391cb34610e6147f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:48:01.648947 10845 tablet_service.cc:1467] Processing CreateTablet for tablet abfca6c37ff2458391cb34610e6147f8 (DEFAULT_TABLE table=test-table [id=b09e720fe7c54782a1ad2f86744afccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:48:01.650332 10845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abfca6c37ff2458391cb34610e6147f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:48:01.651711 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:01.630237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34903 (local address 127.0.0.1:58398)
0929 20:48:01.642965 (+ 12728us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:01.642983 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:48:01.643109 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:48:01.646252 (+ 3143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:48:01.646257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:48:01.646282 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:48:01.646467 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:01.646479 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.647934 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.647939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:01.649255 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:01.649268 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.650949 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.650955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:01.651146 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:01.651188 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 20:48:01.651272 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 20:48:01.651433 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12621,"thread_start_us":14904,"threads_started":1}
I20260929 20:48:01.651764 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:01.630360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58398 (local address 127.0.97.2:34903)
0929 20:48:01.645762 (+ 15402us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:01.645768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:01.645792 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:48:01.645876 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:01.645881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:01.646065 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:01.646238 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:48:01.646633 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.647815 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.649484 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.650790 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.650943 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:01.651162 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:01.651321 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 20:48:01.651475 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15272,"thread_start_us":15178,"threads_started":1}
I20260929 20:48:01.658502 10979 tablet_service.cc:1467] Processing CreateTablet for tablet abfca6c37ff2458391cb34610e6147f8 (DEFAULT_TABLE table=test-table [id=b09e720fe7c54782a1ad2f86744afccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:48:01.659706 10979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abfca6c37ff2458391cb34610e6147f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:48:01.661847 11195 tablet_bootstrap.cc:492] T abfca6c37ff2458391cb34610e6147f8 P fa40516ea7ae46f68f0dea8571e933e2: Bootstrap starting.
I20260929 20:48:01.663787 11196 tablet_bootstrap.cc:492] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb: Bootstrap starting.
I20260929 20:48:01.665721 11195 tablet_bootstrap.cc:654] T abfca6c37ff2458391cb34610e6147f8 P fa40516ea7ae46f68f0dea8571e933e2: Neither blocks nor log segments found. Creating new log.
I20260929 20:48:01.666821 11196 tablet_bootstrap.cc:654] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb: Neither blocks nor log segments found. Creating new log.
I20260929 20:48:01.667119 11195 log.cc:824] T abfca6c37ff2458391cb34610e6147f8 P fa40516ea7ae46f68f0dea8571e933e2: Log is configured to *not* fsync() on all Append() calls
I20260929 20:48:01.667833 11196 log.cc:824] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb: Log is configured to *not* fsync() on all Append() calls
I20260929 20:48:01.669981 11195 tablet_bootstrap.cc:492] T abfca6c37ff2458391cb34610e6147f8 P fa40516ea7ae46f68f0dea8571e933e2: No bootstrap required, opened a new log
I20260929 20:48:01.670248 11195 ts_tablet_manager.cc:1397] T abfca6c37ff2458391cb34610e6147f8 P fa40516ea7ae46f68f0dea8571e933e2: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 20:48:01.670687 11196 tablet_bootstrap.cc:492] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb: No bootstrap required, opened a new log
I20260929 20:48:01.670930 11196 ts_tablet_manager.cc:1397] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 20:48:01.672711 11199 tablet_bootstrap.cc:492] T abfca6c37ff2458391cb34610e6147f8 P 5056dc9e784748499153f5843aeff2f9: Bootstrap starting.
I20260929 20:48:01.676483 11199 tablet_bootstrap.cc:654] T abfca6c37ff2458391cb34610e6147f8 P 5056dc9e784748499153f5843aeff2f9: Neither blocks nor log segments found. Creating new log.
I20260929 20:48:01.677762 11199 log.cc:824] T abfca6c37ff2458391cb34610e6147f8 P 5056dc9e784748499153f5843aeff2f9: Log is configured to *not* fsync() on all Append() calls
I20260929 20:48:01.680624 11199 tablet_bootstrap.cc:492] T abfca6c37ff2458391cb34610e6147f8 P 5056dc9e784748499153f5843aeff2f9: No bootstrap required, opened a new log
I20260929 20:48:01.680907 11199 ts_tablet_manager.cc:1397] T abfca6c37ff2458391cb34610e6147f8 P 5056dc9e784748499153f5843aeff2f9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 20:48:01.680994 11196 raft_consensus.cc:348] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e0d0ce1e5c54fa9b7a816955a035feb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32779 } } peers { permanent_uuid: "fa40516ea7ae46f68f0dea8571e933e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37621 } } peers { permanent_uuid: "5056dc9e784748499153f5843aeff2f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34903 } }
I20260929 20:48:01.680994 11195 raft_consensus.cc:348] T abfca6c37ff2458391cb34610e6147f8 P fa40516ea7ae46f68f0dea8571e933e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e0d0ce1e5c54fa9b7a816955a035feb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32779 } } peers { permanent_uuid: "fa40516ea7ae46f68f0dea8571e933e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37621 } } peers { permanent_uuid: "5056dc9e784748499153f5843aeff2f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34903 } }
I20260929 20:48:01.681463 11196 raft_consensus.cc:374] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:48:01.681463 11195 raft_consensus.cc:374] T abfca6c37ff2458391cb34610e6147f8 P fa40516ea7ae46f68f0dea8571e933e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:48:01.681557 11195 raft_consensus.cc:729] T abfca6c37ff2458391cb34610e6147f8 P fa40516ea7ae46f68f0dea8571e933e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa40516ea7ae46f68f0dea8571e933e2, State: Initialized, Role: FOLLOWER
I20260929 20:48:01.681557 11196 raft_consensus.cc:729] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e0d0ce1e5c54fa9b7a816955a035feb, State: Initialized, Role: FOLLOWER
I20260929 20:48:01.682168 11195 consensus_queue.cc:260] T abfca6c37ff2458391cb34610e6147f8 P fa40516ea7ae46f68f0dea8571e933e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e0d0ce1e5c54fa9b7a816955a035feb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32779 } } peers { permanent_uuid: "fa40516ea7ae46f68f0dea8571e933e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37621 } } peers { permanent_uuid: "5056dc9e784748499153f5843aeff2f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34903 } }
I20260929 20:48:01.682166 11196 consensus_queue.cc:260] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e0d0ce1e5c54fa9b7a816955a035feb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32779 } } peers { permanent_uuid: "fa40516ea7ae46f68f0dea8571e933e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37621 } } peers { permanent_uuid: "5056dc9e784748499153f5843aeff2f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34903 } }
I20260929 20:48:01.683723 11177 heartbeater.cc:503] Master 127.0.97.62:43461 was elected leader, sending a full tablet report...
I20260929 20:48:01.684006 11196 ts_tablet_manager.cc:1428] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260929 20:48:01.684183 11195 ts_tablet_manager.cc:1428] T abfca6c37ff2458391cb34610e6147f8 P fa40516ea7ae46f68f0dea8571e933e2: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260929 20:48:01.690706 11199 raft_consensus.cc:348] T abfca6c37ff2458391cb34610e6147f8 P 5056dc9e784748499153f5843aeff2f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e0d0ce1e5c54fa9b7a816955a035feb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32779 } } peers { permanent_uuid: "fa40516ea7ae46f68f0dea8571e933e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37621 } } peers { permanent_uuid: "5056dc9e784748499153f5843aeff2f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34903 } }
I20260929 20:48:01.691113 11199 raft_consensus.cc:374] T abfca6c37ff2458391cb34610e6147f8 P 5056dc9e784748499153f5843aeff2f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:48:01.691207 11199 raft_consensus.cc:729] T abfca6c37ff2458391cb34610e6147f8 P 5056dc9e784748499153f5843aeff2f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5056dc9e784748499153f5843aeff2f9, State: Initialized, Role: FOLLOWER
I20260929 20:48:01.691776 11199 consensus_queue.cc:260] T abfca6c37ff2458391cb34610e6147f8 P 5056dc9e784748499153f5843aeff2f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e0d0ce1e5c54fa9b7a816955a035feb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32779 } } peers { permanent_uuid: "fa40516ea7ae46f68f0dea8571e933e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37621 } } peers { permanent_uuid: "5056dc9e784748499153f5843aeff2f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34903 } }
I20260929 20:48:01.692727 11044 heartbeater.cc:503] Master 127.0.97.62:43461 was elected leader, sending a full tablet report...
I20260929 20:48:01.693233 11199 ts_tablet_manager.cc:1428] T abfca6c37ff2458391cb34610e6147f8 P 5056dc9e784748499153f5843aeff2f9: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
W20260929 20:48:01.811507 11178 tablet.cc:2369] T abfca6c37ff2458391cb34610e6147f8 P fa40516ea7ae46f68f0dea8571e933e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:48:01.818429 11202 raft_consensus.cc:482] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:48:01.818707 11202 raft_consensus.cc:504] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e0d0ce1e5c54fa9b7a816955a035feb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32779 } } peers { permanent_uuid: "fa40516ea7ae46f68f0dea8571e933e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37621 } } peers { permanent_uuid: "5056dc9e784748499153f5843aeff2f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34903 } }
I20260929 20:48:01.820391 11202 leader_election.cc:288] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa40516ea7ae46f68f0dea8571e933e2 (127.0.97.3:37621), 5056dc9e784748499153f5843aeff2f9 (127.0.97.2:34903)
I20260929 20:48:01.828436 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:01.820966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38007 (local address 127.0.97.2:34903)
0929 20:48:01.821173 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:01.821179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:01.821756 (+ 577us) server_negotiation.cc:407] Connection header received
0929 20:48:01.821871 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:01.821877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:01.821945 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:01.822011 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:48:01.822563 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.823625 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.826137 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.827376 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.827522 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:01.827619 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:01.828034 (+ 415us) server_negotiation.cc:299] Negotiation successful
0929 20:48:01.828162 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 20:48:01.828567 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:01.820670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34903 (local address 127.0.97.1:38007)
0929 20:48:01.821654 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:01.821667 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 20:48:01.821770 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:48:01.822048 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:48:01.822053 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:48:01.822078 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:48:01.822411 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:01.822419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.823751 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.823755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:01.825956 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:01.825971 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.827591 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.827597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:01.827786 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:01.827910 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 20:48:01.827991 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 20:48:01.828158 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"mutex_wait_us":104,"thread_start_us":664,"threads_started":1}
I20260929 20:48:01.828649 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:01.820581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37621 (local address 127.0.97.1:51203)
0929 20:48:01.821357 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:01.821372 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:48:01.821519 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:48:01.821824 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:48:01.821828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:48:01.821850 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:48:01.822178 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:01.822189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.823633 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.823637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:01.825960 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:01.825971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.827643 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.827649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:01.827856 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:01.827897 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 20:48:01.827991 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 20:48:01.828158 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":83,"thread_start_us":497,"threads_started":1}
I20260929 20:48:01.828872 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:01.820675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51203 (local address 127.0.97.3:37621)
0929 20:48:01.820899 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:01.820907 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:01.821453 (+ 546us) server_negotiation.cc:407] Connection header received
0929 20:48:01.821653 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:01.821656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:01.821714 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:01.821793 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:48:01.822342 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.823511 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.826130 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.827469 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.827572 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:01.827659 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:01.828033 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 20:48:01.828635 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 20:48:01.828444 11045 tablet.cc:2369] T abfca6c37ff2458391cb34610e6147f8 P 5056dc9e784748499153f5843aeff2f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:48:01.829752 11132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfca6c37ff2458391cb34610e6147f8" candidate_uuid: "0e0d0ce1e5c54fa9b7a816955a035feb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa40516ea7ae46f68f0dea8571e933e2" is_pre_election: true
I20260929 20:48:01.829856 10999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfca6c37ff2458391cb34610e6147f8" candidate_uuid: "0e0d0ce1e5c54fa9b7a816955a035feb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5056dc9e784748499153f5843aeff2f9" is_pre_election: true
I20260929 20:48:01.830253 11132 raft_consensus.cc:2454] T abfca6c37ff2458391cb34610e6147f8 P fa40516ea7ae46f68f0dea8571e933e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e0d0ce1e5c54fa9b7a816955a035feb in term 0.
I20260929 20:48:01.830292 10999 raft_consensus.cc:2454] T abfca6c37ff2458391cb34610e6147f8 P 5056dc9e784748499153f5843aeff2f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e0d0ce1e5c54fa9b7a816955a035feb in term 0.
I20260929 20:48:01.830883 10799 leader_election.cc:302] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e0d0ce1e5c54fa9b7a816955a035feb, 5056dc9e784748499153f5843aeff2f9; no voters:
I20260929 20:48:01.831339 11202 raft_consensus.cc:2781] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:48:01.831543 11202 raft_consensus.cc:482] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:48:01.831635 11202 raft_consensus.cc:3037] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:48:01.833683 11202 raft_consensus.cc:504] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e0d0ce1e5c54fa9b7a816955a035feb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32779 } } peers { permanent_uuid: "fa40516ea7ae46f68f0dea8571e933e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37621 } } peers { permanent_uuid: "5056dc9e784748499153f5843aeff2f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34903 } }
I20260929 20:48:01.834199 11202 leader_election.cc:288] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb [CANDIDATE]: Term 1 election: Requested vote from peers fa40516ea7ae46f68f0dea8571e933e2 (127.0.97.3:37621), 5056dc9e784748499153f5843aeff2f9 (127.0.97.2:34903)
I20260929 20:48:01.834677 11132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfca6c37ff2458391cb34610e6147f8" candidate_uuid: "0e0d0ce1e5c54fa9b7a816955a035feb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa40516ea7ae46f68f0dea8571e933e2"
I20260929 20:48:01.834673 10999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abfca6c37ff2458391cb34610e6147f8" candidate_uuid: "0e0d0ce1e5c54fa9b7a816955a035feb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5056dc9e784748499153f5843aeff2f9"
I20260929 20:48:01.834867 10999 raft_consensus.cc:3037] T abfca6c37ff2458391cb34610e6147f8 P 5056dc9e784748499153f5843aeff2f9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:48:01.834867 11132 raft_consensus.cc:3037] T abfca6c37ff2458391cb34610e6147f8 P fa40516ea7ae46f68f0dea8571e933e2 [term 0 FOLLOWER]: Advancing to term 1
W20260929 20:48:01.835163 10911 tablet.cc:2369] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:48:01.837881 10999 raft_consensus.cc:2454] T abfca6c37ff2458391cb34610e6147f8 P 5056dc9e784748499153f5843aeff2f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e0d0ce1e5c54fa9b7a816955a035feb in term 1.
I20260929 20:48:01.837881 11132 raft_consensus.cc:2454] T abfca6c37ff2458391cb34610e6147f8 P fa40516ea7ae46f68f0dea8571e933e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e0d0ce1e5c54fa9b7a816955a035feb in term 1.
I20260929 20:48:01.838251 10799 leader_election.cc:302] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e0d0ce1e5c54fa9b7a816955a035feb, 5056dc9e784748499153f5843aeff2f9; no voters:
I20260929 20:48:01.838518 11202 raft_consensus.cc:2781] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:48:01.838935 11202 raft_consensus.cc:686] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb [term 1 LEADER]: Becoming Leader. State: Replica: 0e0d0ce1e5c54fa9b7a816955a035feb, State: Running, Role: LEADER
I20260929 20:48:01.839461 11202 consensus_queue.cc:237] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e0d0ce1e5c54fa9b7a816955a035feb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32779 } } peers { permanent_uuid: "fa40516ea7ae46f68f0dea8571e933e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37621 } } peers { permanent_uuid: "5056dc9e784748499153f5843aeff2f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34903 } }
I20260929 20:48:01.843595 10720 catalog_manager.cc:5495] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e0d0ce1e5c54fa9b7a816955a035feb (127.0.97.1). New cstate: current_term: 1 leader_uuid: "0e0d0ce1e5c54fa9b7a816955a035feb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e0d0ce1e5c54fa9b7a816955a035feb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa40516ea7ae46f68f0dea8571e933e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5056dc9e784748499153f5843aeff2f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34903 } health_report { overall_health: UNKNOWN } } }
I20260929 20:48:01.874759 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:01.869673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33070 (local address 127.0.97.1:32779)
0929 20:48:01.869896 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:01.869901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:01.869921 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:48:01.870009 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:01.870013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:01.870089 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:01.870219 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:48:01.870519 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.871300 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.872579 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.873004 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.873122 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:01.873673 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:48:01.874208 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:48:01.874341 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:01.874459 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 20:48:01.874526 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 20:48:01.882869 11132 raft_consensus.cc:1261] T abfca6c37ff2458391cb34610e6147f8 P fa40516ea7ae46f68f0dea8571e933e2 [term 1 FOLLOWER]: Refusing update from remote peer 0e0d0ce1e5c54fa9b7a816955a035feb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:48:01.882869 10999 raft_consensus.cc:1261] T abfca6c37ff2458391cb34610e6147f8 P 5056dc9e784748499153f5843aeff2f9 [term 1 FOLLOWER]: Refusing update from remote peer 0e0d0ce1e5c54fa9b7a816955a035feb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:48:01.883931 11202 consensus_queue.cc:1035] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa40516ea7ae46f68f0dea8571e933e2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:48:01.884214 11206 consensus_queue.cc:1035] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5056dc9e784748499153f5843aeff2f9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:48:01.895478 11209 mvcc.cc:205] Tried to move back new op lower bound from 7334768156181594112 to 7334768156017774592. Current Snapshot: MvccSnapshot[applied={T|T < 7334768156181594112}]
I20260929 20:48:01.908810 10720 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:48:01.909062 10720 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:48:01.912456 10720 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5443aa91e484a6f95403b7ed5eeea5d: Sending DeleteTablet for 3 replicas of tablet abfca6c37ff2458391cb34610e6147f8
I20260929 20:48:01.913529 11112 tablet_service.cc:1514] Processing DeleteTablet for tablet abfca6c37ff2458391cb34610e6147f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:48:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55492
I20260929 20:48:01.913535 10845 tablet_service.cc:1514] Processing DeleteTablet for tablet abfca6c37ff2458391cb34610e6147f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:48:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33064
I20260929 20:48:01.914248 10979 tablet_service.cc:1514] Processing DeleteTablet for tablet abfca6c37ff2458391cb34610e6147f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:48:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58398
I20260929 20:48:01.914398 11219 tablet_replica.cc:330] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb: stopping tablet replica
I20260929 20:48:01.914913 11219 raft_consensus.cc:2229] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:48:01.915618 11219 raft_consensus.cc:2258] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:48:01.916410 11220 tablet_replica.cc:330] T abfca6c37ff2458391cb34610e6147f8 P fa40516ea7ae46f68f0dea8571e933e2: stopping tablet replica
I20260929 20:48:01.916636 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 10782
I20260929 20:48:01.916893 11220 raft_consensus.cc:2229] T abfca6c37ff2458391cb34610e6147f8 P fa40516ea7ae46f68f0dea8571e933e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:48:01.917403 11220 raft_consensus.cc:2258] T abfca6c37ff2458391cb34610e6147f8 P fa40516ea7ae46f68f0dea8571e933e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:48:01.917771 11219 ts_tablet_manager.cc:1905] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:48:01.919448 11220 ts_tablet_manager.cc:1905] T abfca6c37ff2458391cb34610e6147f8 P fa40516ea7ae46f68f0dea8571e933e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:48:01.924425 11219 ts_tablet_manager.cc:1918] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:48:01.924587 11219 log.cc:1196] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/abfca6c37ff2458391cb34610e6147f8
I20260929 20:48:01.925065 11219 ts_tablet_manager.cc:1939] T abfca6c37ff2458391cb34610e6147f8 P 0e0d0ce1e5c54fa9b7a816955a035feb: Deleting consensus metadata
I20260929 20:48:01.925475 11221 tablet_replica.cc:330] T abfca6c37ff2458391cb34610e6147f8 P 5056dc9e784748499153f5843aeff2f9: stopping tablet replica
I20260929 20:48:01.925910 11221 raft_consensus.cc:2229] T abfca6c37ff2458391cb34610e6147f8 P 5056dc9e784748499153f5843aeff2f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:48:01.926405 11221 raft_consensus.cc:2258] T abfca6c37ff2458391cb34610e6147f8 P 5056dc9e784748499153f5843aeff2f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:48:01.926687 10705 catalog_manager.cc:4841] TS 0e0d0ce1e5c54fa9b7a816955a035feb (127.0.97.1:32779): tablet abfca6c37ff2458391cb34610e6147f8 (table test-table [id=b09e720fe7c54782a1ad2f86744afccb]) successfully deleted
I20260929 20:48:01.927232 11220 ts_tablet_manager.cc:1918] T abfca6c37ff2458391cb34610e6147f8 P fa40516ea7ae46f68f0dea8571e933e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:48:01.927433 11220 log.cc:1196] T abfca6c37ff2458391cb34610e6147f8 P fa40516ea7ae46f68f0dea8571e933e2: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/abfca6c37ff2458391cb34610e6147f8
I20260929 20:48:01.927978 11220 ts_tablet_manager.cc:1939] T abfca6c37ff2458391cb34610e6147f8 P fa40516ea7ae46f68f0dea8571e933e2: Deleting consensus metadata
I20260929 20:48:01.928499 11221 ts_tablet_manager.cc:1905] T abfca6c37ff2458391cb34610e6147f8 P 5056dc9e784748499153f5843aeff2f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:48:01.929688 10708 catalog_manager.cc:4841] TS fa40516ea7ae46f68f0dea8571e933e2 (127.0.97.3:37621): tablet abfca6c37ff2458391cb34610e6147f8 (table test-table [id=b09e720fe7c54782a1ad2f86744afccb]) successfully deleted
I20260929 20:48:01.934537 11221 ts_tablet_manager.cc:1918] T abfca6c37ff2458391cb34610e6147f8 P 5056dc9e784748499153f5843aeff2f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:48:01.934679 11221 log.cc:1196] T abfca6c37ff2458391cb34610e6147f8 P 5056dc9e784748499153f5843aeff2f9: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/abfca6c37ff2458391cb34610e6147f8
I20260929 20:48:01.935087 11221 ts_tablet_manager.cc:1939] T abfca6c37ff2458391cb34610e6147f8 P 5056dc9e784748499153f5843aeff2f9: Deleting consensus metadata
Sep 29 20:48:01 dist-test-slave-blv7 krb5kdc[10676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:48:01.936461 10706 catalog_manager.cc:4841] TS 5056dc9e784748499153f5843aeff2f9 (127.0.97.2:34903): tablet abfca6c37ff2458391cb34610e6147f8 (table test-table [id=b09e720fe7c54782a1ad2f86744afccb]) successfully deleted
I20260929 20:48:01.942086 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:01.917979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33072 (local address 127.0.97.1:32779)
0929 20:48:01.918154 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:01.918162 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:01.925144 (+ 6982us) server_negotiation.cc:407] Connection header received
0929 20:48:01.925422 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:01.925427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:01.925506 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:01.925619 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:48:01.927888 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.929015 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.930764 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:01.931492 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:01.931625 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:01.936338 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:48:01.936354 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:48:01.936372 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:48:01.936413 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:48:01.939352 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:01.940171 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:01.940178 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:01.940180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:01.940327 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:01.940731 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:01.940735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:01.940737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:01.941101 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:48:01.941355 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:01.941620 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 20:48:01.941796 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:48:01.942790 10905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:48:02.048810 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 10782
I20260929 20:48:02.063092 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 10916
Sep 29 20:48:02 dist-test-slave-blv7 krb5kdc[10676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:48:02.077644 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:02.063885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58410 (local address 127.0.97.2:34903)
0929 20:48:02.064074 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:02.064078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:02.064097 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:48:02.064204 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:02.064209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:02.064270 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:02.064353 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:48:02.065505 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:02.066210 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:02.067677 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:02.068118 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:02.068236 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:02.072131 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:48:02.072145 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:48:02.072160 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:48:02.072195 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:48:02.074630 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:02.075538 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:02.075547 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:02.075550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:02.075700 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:02.076004 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:02.076008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:02.076011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:02.076489 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:48:02.076772 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:02.077054 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 20:48:02.077295 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:48:02.078395 11039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:48:02.165661 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 10916
I20260929 20:48:02.179207 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 11049
Sep 29 20:48:02 dist-test-slave-blv7 krb5kdc[10676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:48:02.193289 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:02.180008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55494 (local address 127.0.97.3:37621)
0929 20:48:02.180180 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:02.180186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:02.180223 (+ 37us) server_negotiation.cc:407] Connection header received
0929 20:48:02.180411 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:02.180415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:02.180509 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:02.180592 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:48:02.181825 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:02.182525 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:02.183810 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:02.184198 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:02.184307 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:02.188462 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:48:02.188484 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:48:02.188501 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:48:02.188540 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:48:02.190992 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:02.191699 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:02.191707 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:02.191709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:02.191835 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:02.192125 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:02.192128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:02.192130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:02.192437 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:48:02.192676 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:02.192871 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 20:48:02.193024 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:48:02.193929 11172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:48:02.280174 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 11049
I20260929 20:48:02.295116 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 10689
I20260929 20:48:02.296211 10751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:48:02.398844 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 10689
2026-09-29T20:48:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4039 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:48:02 dist-test-slave-blv7 krb5kdc[11229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:48:02 dist-test-slave-blv7 krb5kdc[11229](info): set up 2 sockets
Sep 29 20:48:02 dist-test-slave-blv7 krb5kdc[11229](info): commencing operation
krb5kdc: starting...
W20260929 20:48:03.419682 388 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:48:03 dist-test-slave-blv7 krb5kdc[11229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:48:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:48:03Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260929 20:48:03.722303 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskf3T6XB/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/postgres -l logfile start
2026-09-29 20:48:06.141 UTC [11259] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 20:48:06.141 UTC [11259] LOG: listening on IPv4 address "127.0.97.20", port 54075
2026-09-29 20:48:06.145 UTC [11259] LOG: listening on Unix socket "/tmp/.s.PGSQL.54075"
2026-09-29 20:48:06.165 UTC [11262] LOG: database system was shut down at 2026-09-29 20:48:04 UTC
2026-09-29 20:48:06.168 UTC [11259] LOG: database system is ready to accept connections
I20260929 20:48:07.051231 388 mini_postgres.cc:96] Postgres bound to 54075
2026-09-29 20:48:07.066 UTC [11270] FATAL: database "slave" does not exist
127.0.97.20:54075 - accepting connections
I20260929 20:48:07.067255 388 mini_ranger.cc:161] Starting Ranger...
I20260929 20:48:07.089965 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 20:48:07.569873 388 mini_ranger.cc:87] Created ranger Postgres database
I20260929 20:48:07.569983 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ranger-admin
2026-09-29 20:48:07,842 [I] DB FLAVOR :POSTGRES
2026-09-29 20:48:07,843 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 20:48:07,843 [I] Checking connection..
2026-09-29 20:48:07,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 20:48:08,077 [I] Checking connection passed.
2026-09-29 20:48:08,077 [I] --------- Verifying version history table ---------
2026-09-29 20:48:08,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:48:08,299 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 20:48:08,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:48:08,523 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 20:48:08,524 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 20:48:08,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 20:48:08.751 UTC [11334] WARNING: there is no transaction in progress
2026-09-29 20:48:08,757 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 20:48:08,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:48:08,991 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 20:48:08,992 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 20:48:08,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 20:48:09,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 20:48:09,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-blv7.c.gcp-upstream.internal','N') ;"
2026-09-29 20:48:09,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 20:48:09,960 [I] Table x_portal_user does not exist in database ranger
2026-09-29 20:48:09,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 20:48:10,190 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 20:48:10,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 20:48:10,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 20:48:10,680 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 20:48:10,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 20:48:11.642 UTC [11442] WARNING: there is no transaction in progress
2026-09-29 20:48:11.659 UTC [11442] WARNING: there is no transaction in progress
2026-09-29 20:48:11.676 UTC [11442] WARNING: there is no transaction in progress
2026-09-29 20:48:11.692 UTC [11442] WARNING: there is no transaction in progress
2026-09-29 20:48:11.708 UTC [11442] WARNING: there is no transaction in progress
2026-09-29 20:48:11.725 UTC [11442] WARNING: there is no transaction in progress
2026-09-29 20:48:11.778 UTC [11442] WARNING: there is no transaction in progress
2026-09-29 20:48:11.784 UTC [11442] WARNING: there is no transaction in progress
2026-09-29 20:48:11.798 UTC [11442] WARNING: there is no transaction in progress
2026-09-29 20:48:11.808 UTC [11442] WARNING: there is no transaction in progress
2026-09-29 20:48:11.818 UTC [11442] WARNING: there is no transaction in progress
2026-09-29 20:48:11.831 UTC [11442] WARNING: there is no transaction in progress
2026-09-29 20:48:11.840 UTC [11442] WARNING: there is no transaction in progress
2026-09-29 20:48:11.846 UTC [11442] WARNING: there is no transaction in progress
2026-09-29 20:48:11.852 UTC [11442] WARNING: there is no transaction in progress
2026-09-29 20:48:11.859 UTC [11442] WARNING: there is no transaction in progress
2026-09-29 20:48:12,468 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 20:48:12,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 20:48:12,693 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 20:48:12,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 20:48:12,950 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 20:48:12,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 20:48:13,177 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 20:48:13,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 20:48:13,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 20:48:13,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-blv7.c.gcp-upstream.internal';"
2026-09-29 20:48:13,906 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 20:48:13,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 20:48:14,135 [I] DB_PATCHES have already been applied
I20260929 20:48:14.140847 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskf3T6XB/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 20:48:14.140933 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260929 20:48:14.154204 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:58521
Sep 29, 2026 8:48:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 8:48:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 8:48:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 8:48:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 8:48:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 8:48:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 8:48:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 20:48:15.161933 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.095s user 0.001s sys 0.032s
Sep 29, 2026 8:48:33 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 8:48:33 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 8:48:33 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 8:48:33 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 8:48:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 8:48:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 8:48:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 8:48:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 8:48:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 8:48:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 8:48:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 8:48:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 8:48:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 8:48:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 8:48:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 8:48:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 8:48:33 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 8:48:34 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 20:48:34.966804 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 20:48:35 dist-test-slave-blv7 krb5kdc[11229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714915, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 20:48:35.133457 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 20:48:35.158341 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 20:48:36.058308 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 20:48:36.058451 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ranger-kms
I20260929 20:48:36.058537 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:54075
2026-09-29 20:48:36,444 [I] DB FLAVOR :POSTGRES
2026-09-29 20:48:36,444 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 20:48:36,444 [I] Checking connection
2026-09-29 20:48:36,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 20:48:36,674 [I] connection success
2026-09-29 20:48:36,674 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 20:48:36,674 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 20:48:36,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 20:48:36,906 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 20:48:36,906 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 20:48:36,907 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 20:48:36,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54075/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 20:48:37,153 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 20:48:37.579452 388 mini_ranger_kms.cc:310] Created kms service
I20260929 20:48:37.802423 388 mini_ranger_kms.cc:326] Created kudu user
I20260929 20:48:37.847695 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 20:48:37 dist-test-slave-blv7 krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714915, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:48:38.230520 388 mini_ranger_kms.cc:384] Added ranger policy
I20260929 20:48:38.230850 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 20:48:38.230899 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260929 20:48:38.251242 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11622
I20260929 20:48:38.251399 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:60569
Sep 29, 2026 8:48:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 8:48:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 8:48:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 8:48:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 8:48:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 8:48:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 20:48:40.334688 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.201s user 0.109s sys 0.293s
I20260929 20:48:40.334957 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 20:48:40.335198 388 mini_ranger_kms.cc:401] 127.0.97.20:60569/kms/v1/keys
Sep 29 20:48:40 dist-test-slave-blv7 krb5kdc[11229](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790714920, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 20:48:40 dist-test-slave-blv7 krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714920, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 20:48:41 dist-test-slave-blv7 krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714920, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 20:48:43 dist-test-slave-blv7 krb5kdc[11229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:48:43.558990 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42347
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42347
--ranger_config_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60569/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:48:43.886562 11673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:48:43.886835 11673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:48:43.886945 11673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:48:43.894075 11673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:48:43.894178 11673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:48:43.894232 11673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:48:43.894305 11673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:48:43.894361 11673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:48:43.904573 11673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:60569/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42347
--ranger_config_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:48:43.905849 11673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:48:43.907208 11673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:48:43.923213 11678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:48:43.923271 11681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:48:43.923463 11679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:48:43.924320 11673 server_base.cc:1034] running on GCE node
I20260929 20:48:43.925083 11673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:48:43.926461 11673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:48:43.927676 11673 hybrid_clock.cc:648] HybridClock initialized: now 1790714923927627 us; error 44 us; skew 500 ppm
I20260929 20:48:43.928002 11673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:48:43 dist-test-slave-blv7 krb5kdc[11229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:48:43.930878 11673 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:48:43.932318 11673 webserver.cc:460] Webserver started at http://127.0.97.62:33987/ using document root <none> and password file <none>
I20260929 20:48:43.932955 11673 fs_manager.cc:359] Metadata directory not provided
I20260929 20:48:43.933043 11673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:48:43.933279 11673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:48:43 dist-test-slave-blv7 krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:48:43.982239 11673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "eefce229379c4076b3082745c55118a3"
format_stamp: "Formatted at 2026-09-29 20:48:43 on dist-test-slave-blv7"
server_key: "9205bd4121371b2c55770d8d6a4fe2bd"
server_key_iv: "813e2d67e208ca302b2d35faa9aebcb6"
server_key_version: "kuduclusterkey@0"
I20260929 20:48:43.983889 11673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "eefce229379c4076b3082745c55118a3"
format_stamp: "Formatted at 2026-09-29 20:48:43 on dist-test-slave-blv7"
server_key: "9205bd4121371b2c55770d8d6a4fe2bd"
server_key_iv: "813e2d67e208ca302b2d35faa9aebcb6"
server_key_version: "kuduclusterkey@0"
I20260929 20:48:43.989145 11673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:48:44.008419 11688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:48:44.009868 11673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:48:44.010028 11673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "eefce229379c4076b3082745c55118a3"
format_stamp: "Formatted at 2026-09-29 20:48:43 on dist-test-slave-blv7"
server_key: "9205bd4121371b2c55770d8d6a4fe2bd"
server_key_iv: "813e2d67e208ca302b2d35faa9aebcb6"
server_key_version: "kuduclusterkey@0"
I20260929 20:48:44.010233 11673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:48:44.021231 11673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:48:44.025365 11673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:48:44.025635 11673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:48:44.045744 11673 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42347
I20260929 20:48:44.046212 11739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42347 every 8 connection(s)
I20260929 20:48:44.047571 11673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:48:44.051426 11740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:48:44.056656 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 11673
I20260929 20:48:44.056854 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:48:44.057614 388 external_mini_cluster.cc:1428] Setting key b82f976b0b1d31067f5d27a74065c897
I20260929 20:48:44.063743 11740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3: Bootstrap starting.
I20260929 20:48:44.068464 11740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3: Neither blocks nor log segments found. Creating new log.
I20260929 20:48:44.070777 11740 log.cc:824] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3: Log is configured to *not* fsync() on all Append() calls
I20260929 20:48:44.075337 11740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3: No bootstrap required, opened a new log
Sep 29 20:48:44 dist-test-slave-blv7 krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:48:44.086627 11740 raft_consensus.cc:348] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefce229379c4076b3082745c55118a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42347 } }
I20260929 20:48:44.087162 11740 raft_consensus.cc:374] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:48:44.087272 11740 raft_consensus.cc:729] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eefce229379c4076b3082745c55118a3, State: Initialized, Role: FOLLOWER
I20260929 20:48:44.087996 11740 consensus_queue.cc:260] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefce229379c4076b3082745c55118a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42347 } }
I20260929 20:48:44.088277 11740 raft_consensus.cc:388] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:48:44.088459 11740 raft_consensus.cc:482] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:48:44.088634 11740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:48:44.090934 11740 raft_consensus.cc:504] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefce229379c4076b3082745c55118a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42347 } }
I20260929 20:48:44.091466 11740 leader_election.cc:302] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eefce229379c4076b3082745c55118a3; no voters:
I20260929 20:48:44.092098 11740 leader_election.cc:288] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:48:44.092272 11745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:48:44.092559 11743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:44.059731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49560 (local address 127.0.97.62:42347)
0929 20:48:44.071572 (+ 11841us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:44.071589 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:44.072203 (+ 614us) server_negotiation.cc:407] Connection header received
0929 20:48:44.074257 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:44.074285 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:44.074708 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:44.075149 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:48:44.076587 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:44.077979 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:44.079333 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:44.080087 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:44.080263 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:44.084113 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:48:44.084122 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:48:44.084138 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:48:44.084186 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:48:44.087806 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:44.088543 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:44.088552 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:44.088562 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:44.088742 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:44.089212 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:44.089218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:44.089222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:44.089848 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:48:44.090166 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:44.090689 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 20:48:44.091142 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11587,"thread_start_us":11351,"threads_started":1}
I20260929 20:48:44.093613 11745 raft_consensus.cc:686] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3 [term 1 LEADER]: Becoming Leader. State: Replica: eefce229379c4076b3082745c55118a3, State: Running, Role: LEADER
I20260929 20:48:44.094137 11740 sys_catalog.cc:564] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:48:44.094182 11745 consensus_queue.cc:237] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefce229379c4076b3082745c55118a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42347 } }
I20260929 20:48:44.100929 11747 sys_catalog.cc:455] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eefce229379c4076b3082745c55118a3. Latest consensus state: current_term: 1 leader_uuid: "eefce229379c4076b3082745c55118a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefce229379c4076b3082745c55118a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42347 } } }
I20260929 20:48:44.101006 11746 sys_catalog.cc:455] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eefce229379c4076b3082745c55118a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefce229379c4076b3082745c55118a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42347 } } }
I20260929 20:48:44.101581 11747 sys_catalog.cc:458] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3 [sys.catalog]: This master's current role is: LEADER
I20260929 20:48:44.101635 11746 sys_catalog.cc:458] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3 [sys.catalog]: This master's current role is: LEADER
I20260929 20:48:44.109776 11740 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 20:48:44.111446 11752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:48:44.123091 11752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:48:44.142596 11752 catalog_manager.cc:1347] Generated new cluster ID: cc7a6c561e904ecab612df4760228cd2
I20260929 20:48:44.142730 11752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:48:44.152438 11752 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:48:44.154805 11752 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:48:44.165030 11752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3: Generated new TSK 0
I20260929 20:48:44.166095 11752 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:48:44.882961 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60569/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42347
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:48:45 dist-test-slave-blv7 krb5kdc[11229](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790714925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 20:48:45.279858 11785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:48:45.280138 11785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:48:45.280294 11785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:48:45.287712 11785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:48:45.287825 11785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:48:45.287989 11785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:48:45.298352 11785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:60569/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:48:45.299731 11785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:48:45.300956 11785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:48:45.315673 11795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:48:45.317492 11796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:48:45.323964 11799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:48:45.324921 11785 server_base.cc:1034] running on GCE node
I20260929 20:48:45.325611 11785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:48:45.326668 11785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:48:45.327914 11785 hybrid_clock.cc:648] HybridClock initialized: now 1790714925327858 us; error 55 us; skew 500 ppm
I20260929 20:48:45.328238 11785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:48:45 dist-test-slave-blv7 krb5kdc[11229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:48:45.331539 11785 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:48:45.336308 11785 webserver.cc:460] Webserver started at http://127.0.97.1:39325/ using document root <none> and password file <none>
I20260929 20:48:45.337028 11785 fs_manager.cc:359] Metadata directory not provided
I20260929 20:48:45.337141 11785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:48:45.337466 11785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:48:45 dist-test-slave-blv7 krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:48:45.356372 11785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "df07fcd6ccb940798a6d96677c4a9945"
format_stamp: "Formatted at 2026-09-29 20:48:45 on dist-test-slave-blv7"
server_key: "974fd3066ac3fbd455441fc264df6fa9"
server_key_iv: "ad235148472e0dea2f4c4fa6411248c9"
server_key_version: "kuduclusterkey@0"
I20260929 20:48:45.357592 11785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "df07fcd6ccb940798a6d96677c4a9945"
format_stamp: "Formatted at 2026-09-29 20:48:45 on dist-test-slave-blv7"
server_key: "974fd3066ac3fbd455441fc264df6fa9"
server_key_iv: "ad235148472e0dea2f4c4fa6411248c9"
server_key_version: "kuduclusterkey@0"
I20260929 20:48:45.380949 11785 fs_manager.cc:691] Time spent creating directory manager: real 0.023s user 0.007s sys 0.012s
I20260929 20:48:45.411600 11805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:48:45.413534 11785 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.000s sys 0.008s
I20260929 20:48:45.413741 11785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "df07fcd6ccb940798a6d96677c4a9945"
format_stamp: "Formatted at 2026-09-29 20:48:45 on dist-test-slave-blv7"
server_key: "974fd3066ac3fbd455441fc264df6fa9"
server_key_iv: "ad235148472e0dea2f4c4fa6411248c9"
server_key_version: "kuduclusterkey@0"
I20260929 20:48:45.414019 11785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:48:45.441100 11785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:48:45.446534 11785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:48:45.446923 11785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:48:45.448139 11785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:48:45.450016 11785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:48:45.450115 11785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:48:45.450241 11785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:48:45.450305 11785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:48:45.519598 11785 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44827
I20260929 20:48:45.519748 11917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44827 every 8 connection(s)
I20260929 20:48:45.521325 11785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:48:45.533398 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 11785
I20260929 20:48:45.533664 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:48:45.534565 388 external_mini_cluster.cc:1428] Setting key bd65f92c40e9d1fe7f6e35e84ef54583
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Sep 29 20:48:45 dist-test-slave-blv7 krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:48:45.594138 11920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:45.531395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58047 (local address 127.0.97.62:42347)
0929 20:48:45.546647 (+ 15252us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:45.546655 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:45.555674 (+ 9019us) server_negotiation.cc:407] Connection header received
0929 20:48:45.559385 (+ 3711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:45.559391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:45.559491 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:45.559593 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:48:45.563660 (+ 4067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:45.564841 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:45.568694 (+ 3853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:45.570173 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:45.570308 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:45.581537 (+ 11229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:48:45.581544 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:48:45.581549 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:48:45.581592 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:48:45.586802 (+ 5210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:45.591415 (+ 4613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:45.591422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:45.591426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:45.591585 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:45.592095 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:45.592100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:45.592103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:45.592460 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:48:45.592585 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:45.593546 (+ 961us) server_negotiation.cc:299] Negotiation successful
0929 20:48:45.593781 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15079,"thread_start_us":15258,"threads_started":1}
I20260929 20:48:45.596330 11921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:45.532336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42347 (local address 127.0.97.1:58047)
0929 20:48:45.555440 (+ 23104us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:45.555510 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 20:48:45.558872 (+ 3362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:48:45.560197 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:48:45.560210 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:48:45.562191 (+ 1981us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:48:45.563417 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:45.563445 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:45.565385 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:45.565391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:45.568471 (+ 3080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:45.568486 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:45.570474 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:45.570483 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:45.570916 (+ 433us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:45.576366 (+ 5450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:48:45.576411 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:48:45.581190 (+ 4779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:48:45.590387 (+ 9197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:48:45.590403 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:48:45.590424 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:48:45.591251 (+ 827us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:48:45.591729 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:48:45.591734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:48:45.591738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:48:45.591976 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:48:45.592676 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:48:45.592684 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:48:45.593114 (+ 430us) client_negotiation.cc:769] Sending connection context
0929 20:48:45.594425 (+ 1311us) client_negotiation.cc:240] Negotiation successful
0929 20:48:45.594964 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22692,"thread_start_us":26522,"threads_started":1}
I20260929 20:48:45.598402 11918 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42347
I20260929 20:48:45.598712 11918 heartbeater.cc:464] Registering TS with master...
I20260929 20:48:45.599542 11918 heartbeater.cc:511] Master 127.0.97.62:42347 requested a full tablet report, sending...
I20260929 20:48:45.602072 11705 ts_manager.cc:194] Registered new tserver with Master: df07fcd6ccb940798a6d96677c4a9945 (127.0.97.1:44827)
I20260929 20:48:45.604374 11705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:48:45.606662 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60569/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42347
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:48:45 dist-test-slave-blv7 krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 20:48:45 dist-test-slave-blv7 krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260929 20:48:46.131989 11924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:48:46.132439 11924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:48:46.132601 11924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:48:46.148805 11924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:48:46.148977 11924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:48:46.149178 11924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:48:46.165282 11924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:60569/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:48:46.167124 11924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:48:46.168802 11924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:48:46.207329 11935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:48:46.207669 11932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:48:46.208110 11924 server_base.cc:1034] running on GCE node
W20260929 20:48:46.211364 11931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:48:46.212332 11924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:48:46.223356 11924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:48:46.235388 11924 hybrid_clock.cc:648] HybridClock initialized: now 1790714926235258 us; error 131 us; skew 500 ppm
I20260929 20:48:46.235880 11924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:48:46 dist-test-slave-blv7 krb5kdc[11229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:48:46.241911 11924 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:48:46.275458 11924 webserver.cc:460] Webserver started at http://127.0.97.2:37963/ using document root <none> and password file <none>
I20260929 20:48:46.276546 11924 fs_manager.cc:359] Metadata directory not provided
I20260929 20:48:46.276713 11924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:48:46.277120 11924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:48:46 dist-test-slave-blv7 krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:48:46.301754 11924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "07623714add144c6811dc479af030fad"
format_stamp: "Formatted at 2026-09-29 20:48:46 on dist-test-slave-blv7"
server_key: "762fd79136df15a78b0f7a6baf6d3b69"
server_key_iv: "1db20519085c48df81eab8c2d991754b"
server_key_version: "kuduclusterkey@0"
I20260929 20:48:46.302888 11924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "07623714add144c6811dc479af030fad"
format_stamp: "Formatted at 2026-09-29 20:48:46 on dist-test-slave-blv7"
server_key: "762fd79136df15a78b0f7a6baf6d3b69"
server_key_iv: "1db20519085c48df81eab8c2d991754b"
server_key_version: "kuduclusterkey@0"
I20260929 20:48:46.316368 11924 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.007s sys 0.004s
I20260929 20:48:46.349901 11943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:48:46.351804 11924 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.001s sys 0.004s
I20260929 20:48:46.352005 11924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "07623714add144c6811dc479af030fad"
format_stamp: "Formatted at 2026-09-29 20:48:46 on dist-test-slave-blv7"
server_key: "762fd79136df15a78b0f7a6baf6d3b69"
server_key_iv: "1db20519085c48df81eab8c2d991754b"
server_key_version: "kuduclusterkey@0"
I20260929 20:48:46.352245 11924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:48:46.410094 11924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:48:46.415704 11924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:48:46.415990 11924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:48:46.419131 11924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:48:46.421294 11924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:48:46.421384 11924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:48:46.421500 11924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:48:46.421552 11924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:48:46.483321 11924 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35965
I20260929 20:48:46.483359 12060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35965 every 8 connection(s)
I20260929 20:48:46.484879 11924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:48:46.492638 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 11924
I20260929 20:48:46.492867 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:48:46.493407 388 external_mini_cluster.cc:1428] Setting key 5c05fdbb1cf53f8da125504185471143
Sep 29 20:48:46 dist-test-slave-blv7 krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:48:46.525125 12064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:46.488212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42347 (local address 127.0.97.2:34155)
0929 20:48:46.503613 (+ 15401us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:46.503679 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 20:48:46.506046 (+ 2367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:48:46.507149 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:48:46.507162 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:48:46.507760 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:48:46.508549 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:46.508587 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:46.510109 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:46.510115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:46.512137 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:46.512153 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:46.513323 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:46.513330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:46.513638 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:46.514417 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:48:46.514447 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:48:46.517088 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:48:46.520814 (+ 3726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:48:46.520824 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:48:46.520841 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:48:46.521387 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:48:46.521798 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:48:46.521803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:48:46.521807 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:48:46.522020 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:48:46.522682 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:48:46.522689 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:48:46.523088 (+ 399us) client_negotiation.cc:769] Sending connection context
0929 20:48:46.523354 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 20:48:46.523852 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15067,"thread_start_us":14872,"threads_started":1}
I20260929 20:48:46.526322 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:46.487732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34155 (local address 127.0.97.62:42347)
0929 20:48:46.503011 (+ 15279us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:46.504181 (+ 1170us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:46.504208 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:48:46.506500 (+ 2292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:46.506505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:46.506591 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:46.506682 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:48:46.508763 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:46.509924 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:46.512311 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:46.512910 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:46.513028 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:46.517333 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:48:46.517339 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:48:46.517344 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:48:46.517382 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:48:46.520654 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:46.521523 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:46.521529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:46.521532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:46.521683 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:46.522142 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:46.522147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:46.522151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:46.522478 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:48:46.522617 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:46.525775 (+ 3158us) server_negotiation.cc:299] Negotiation successful
0929 20:48:46.525989 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15126,"thread_start_us":16476,"threads_started":1}
I20260929 20:48:46.527667 12061 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42347
I20260929 20:48:46.527953 12061 heartbeater.cc:464] Registering TS with master...
I20260929 20:48:46.529170 12061 heartbeater.cc:511] Master 127.0.97.62:42347 requested a full tablet report, sending...
I20260929 20:48:46.530733 11705 ts_manager.cc:194] Registered new tserver with Master: 07623714add144c6811dc479af030fad (127.0.97.2:35965)
I20260929 20:48:46.531756 11705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34155
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:48:46.574506 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:60569/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42347
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:48:46.608175 11918 heartbeater.cc:503] Master 127.0.97.62:42347 was elected leader, sending a full tablet report...
W20260929 20:48:46.917542 12067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:48:46.917801 12067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:48:46.917914 12067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:48:46.925277 12067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:48:46.925380 12067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:48:46.925516 12067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:48:46.935050 12067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:60569/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:48:46.936748 12067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:48:46.937978 12067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:48:46.950284 12073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:48:46.950255 12072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:48:46.950331 12075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:48:46.951095 12067 server_base.cc:1034] running on GCE node
I20260929 20:48:46.951617 12067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:48:46.952904 12067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:48:46.954098 12067 hybrid_clock.cc:648] HybridClock initialized: now 1790714926954063 us; error 29 us; skew 500 ppm
I20260929 20:48:46.954380 12067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:48:46 dist-test-slave-blv7 krb5kdc[11229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:48:46.957240 12067 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:48:46.958596 12067 webserver.cc:460] Webserver started at http://127.0.97.3:45845/ using document root <none> and password file <none>
I20260929 20:48:46.959252 12067 fs_manager.cc:359] Metadata directory not provided
I20260929 20:48:46.959347 12067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:48:46.959587 12067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:48:46 dist-test-slave-blv7 krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:48:46.972506 12067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "d8dae717db5344d380f882ece85da754"
format_stamp: "Formatted at 2026-09-29 20:48:46 on dist-test-slave-blv7"
server_key: "89e471829c6107cce726926d21f8a8a2"
server_key_iv: "aae73c46fd9aee98ef72a18e8f64c618"
server_key_version: "kuduclusterkey@0"
I20260929 20:48:46.973310 12067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d8dae717db5344d380f882ece85da754"
format_stamp: "Formatted at 2026-09-29 20:48:46 on dist-test-slave-blv7"
server_key: "89e471829c6107cce726926d21f8a8a2"
server_key_iv: "aae73c46fd9aee98ef72a18e8f64c618"
server_key_version: "kuduclusterkey@0"
I20260929 20:48:46.978446 12067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:48:46.993806 12082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:48:46.995321 12067 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:48:46.995465 12067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "d8dae717db5344d380f882ece85da754"
format_stamp: "Formatted at 2026-09-29 20:48:46 on dist-test-slave-blv7"
server_key: "89e471829c6107cce726926d21f8a8a2"
server_key_iv: "aae73c46fd9aee98ef72a18e8f64c618"
server_key_version: "kuduclusterkey@0"
I20260929 20:48:46.995713 12067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:48:47.012466 12067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:48:47.016355 12067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:48:47.016583 12067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:48:47.017414 12067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:48:47.018625 12067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:48:47.018703 12067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:48:47.018790 12067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:48:47.018836 12067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:48:47.050078 12067 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37211
I20260929 20:48:47.050125 12194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37211 every 8 connection(s)
I20260929 20:48:47.051590 12067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:48:47.060472 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 12067
I20260929 20:48:47.060629 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:48:47.061021 388 external_mini_cluster.cc:1428] Setting key a3ce5ba8b64b2de6cd0cb8470bd28288
Sep 29 20:48:47 dist-test-slave-blv7 krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:48:47.070648 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:47.054077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39011 (local address 127.0.97.62:42347)
0929 20:48:47.054521 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:47.054525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:47.055585 (+ 1060us) server_negotiation.cc:407] Connection header received
0929 20:48:47.057711 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:47.057717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:47.057811 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:47.057930 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:48:47.059342 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.060054 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.061692 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.062108 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.062231 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:47.065654 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:48:47.065659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:48:47.065664 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:48:47.065694 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:48:47.068177 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:47.068879 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:47.068883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:47.068885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:47.069001 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:47.069349 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:47.069351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:47.069353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:47.069592 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:48:47.069701 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:47.070259 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 20:48:47.070408 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":217,"threads_started":1}
I20260929 20:48:47.071659 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:47.054695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42347 (local address 127.0.97.3:39011)
0929 20:48:47.055441 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:47.055478 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:48:47.057458 (+ 1980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:48:47.058181 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:48:47.058190 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:48:47.058588 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:48:47.059200 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:47.059221 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.060221 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.060226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:47.061549 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:47.061561 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.062312 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.062319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:47.062599 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:47.063305 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:48:47.063333 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:48:47.065470 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:48:47.068301 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:48:47.068308 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:48:47.068321 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:48:47.068786 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:48:47.069074 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:48:47.069077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:48:47.069080 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:48:47.069267 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:48:47.069680 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:48:47.069685 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:48:47.069965 (+ 280us) client_negotiation.cc:769] Sending connection context
0929 20:48:47.070183 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 20:48:47.070544 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":333,"threads_started":1}
I20260929 20:48:47.072985 12195 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42347
I20260929 20:48:47.073266 12195 heartbeater.cc:464] Registering TS with master...
I20260929 20:48:47.073963 12195 heartbeater.cc:511] Master 127.0.97.62:42347 requested a full tablet report, sending...
I20260929 20:48:47.075419 11705 ts_manager.cc:194] Registered new tserver with Master: d8dae717db5344d380f882ece85da754 (127.0.97.3:37211)
I20260929 20:48:47.076361 11705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39011
I20260929 20:48:47.076558 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:48:47.094338 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:47.081665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37924 (local address 127.0.97.62:42347)
0929 20:48:47.081826 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:47.081831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:47.082246 (+ 415us) server_negotiation.cc:407] Connection header received
0929 20:48:47.082373 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:47.082378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:47.082468 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:47.082594 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:48:47.083957 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.084691 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.086059 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.086499 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.086678 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:47.089088 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:48:47.089093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:48:47.089096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:48:47.089129 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:48:47.091658 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:47.092311 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:47.092316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:47.092318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:47.092457 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:47.092953 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:47.092957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:47.092959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:47.093243 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:48:47.093480 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:47.093789 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 20:48:47.094020 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:48:47.098398 11705 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:48:47.105885 11705 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:48:47.138234 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:47.118064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58086 (local address 127.0.97.3:37211)
0929 20:48:47.130562 (+ 12498us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:47.130571 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:47.130607 (+ 36us) server_negotiation.cc:407] Connection header received
0929 20:48:47.130705 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:47.130714 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:47.130989 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:47.131942 (+ 953us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:48:47.132189 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.133616 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.136048 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.137045 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.137166 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:47.137389 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:47.137760 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 20:48:47.137923 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12306,"thread_start_us":11664,"threads_started":1}
I20260929 20:48:47.138278 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:47.118048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37211 (local address 127.0.0.1:58086)
0929 20:48:47.129273 (+ 11225us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:47.129288 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:48:47.129383 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:48:47.131260 (+ 1877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:48:47.131268 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:48:47.131311 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:48:47.131735 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:47.131745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.133758 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.133764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:47.135915 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:47.135925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.137247 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.137254 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:47.137501 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:47.137558 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 20:48:47.137704 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 20:48:47.137889 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11128,"mutex_wait_us":29,"thread_start_us":11370,"threads_started":1}
I20260929 20:48:47.140117 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:47.118047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44827 (local address 127.0.0.1:44116)
0929 20:48:47.128545 (+ 10498us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:47.128591 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:48:47.128776 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:48:47.132827 (+ 4051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:48:47.132833 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:48:47.132857 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:48:47.133183 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:47.133194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.135309 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.135313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:47.137192 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:47.137207 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.138932 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.138938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:47.139118 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:47.139148 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:48:47.139226 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 20:48:47.139356 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10219,"thread_start_us":10042,"threads_started":1}
I20260929 20:48:47.140143 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:47.118650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44116 (local address 127.0.97.1:44827)
0929 20:48:47.132058 (+ 13408us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:47.132066 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:47.132092 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:48:47.132181 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:47.132188 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:47.132423 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:47.133360 (+ 937us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:48:47.133568 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.135187 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.137373 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.138685 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.138849 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:47.139095 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:47.139296 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 20:48:47.139444 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13226,"thread_start_us":13866,"threads_started":1}
I20260929 20:48:47.142560 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:47.130081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35965 (local address 127.0.0.1:39410)
0929 20:48:47.133579 (+ 3498us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:47.133596 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:48:47.133709 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:48:47.134449 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:48:47.134453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:48:47.134475 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:48:47.134924 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:47.134937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.136841 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.136846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:47.138942 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:47.138954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.141872 (+ 2918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.141879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:47.142038 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:47.142071 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:48:47.142154 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 20:48:47.142278 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3403,"thread_start_us":3286,"threads_started":1}
I20260929 20:48:47.143256 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:47.129348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39410 (local address 127.0.97.2:35965)
0929 20:48:47.133352 (+ 4004us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:47.133361 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:47.133706 (+ 345us) server_negotiation.cc:407] Connection header received
0929 20:48:47.133833 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:47.133839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:47.134046 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:47.134237 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:48:47.135175 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.136690 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.139138 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.141677 (+ 2539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.141811 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:47.141984 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:47.142793 (+ 809us) server_negotiation.cc:299] Negotiation successful
0929 20:48:47.142960 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3855,"thread_start_us":3727,"threads_started":1}
I20260929 20:48:47.142987 12130 tablet_service.cc:1467] Processing CreateTablet for tablet 5a13b4d29ea54a8aa8ebde8145657b25 (DEFAULT_TABLE table=test-table [id=da482bbe2d6b46d3b3afe7ee79f1dcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:48:47.144655 12130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a13b4d29ea54a8aa8ebde8145657b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:48:47.143929 11853 tablet_service.cc:1467] Processing CreateTablet for tablet 5a13b4d29ea54a8aa8ebde8145657b25 (DEFAULT_TABLE table=test-table [id=da482bbe2d6b46d3b3afe7ee79f1dcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:48:47.147413 11996 tablet_service.cc:1467] Processing CreateTablet for tablet 5a13b4d29ea54a8aa8ebde8145657b25 (DEFAULT_TABLE table=test-table [id=da482bbe2d6b46d3b3afe7ee79f1dcd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:48:47.148150 11853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a13b4d29ea54a8aa8ebde8145657b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:48:47.148674 11996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a13b4d29ea54a8aa8ebde8145657b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:48:47.158926 12214 tablet_bootstrap.cc:492] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945: Bootstrap starting.
I20260929 20:48:47.162324 12214 tablet_bootstrap.cc:654] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945: Neither blocks nor log segments found. Creating new log.
I20260929 20:48:47.163230 12215 tablet_bootstrap.cc:492] T 5a13b4d29ea54a8aa8ebde8145657b25 P d8dae717db5344d380f882ece85da754: Bootstrap starting.
I20260929 20:48:47.163619 12214 log.cc:824] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945: Log is configured to *not* fsync() on all Append() calls
I20260929 20:48:47.164781 12216 tablet_bootstrap.cc:492] T 5a13b4d29ea54a8aa8ebde8145657b25 P 07623714add144c6811dc479af030fad: Bootstrap starting.
I20260929 20:48:47.167048 12215 tablet_bootstrap.cc:654] T 5a13b4d29ea54a8aa8ebde8145657b25 P d8dae717db5344d380f882ece85da754: Neither blocks nor log segments found. Creating new log.
I20260929 20:48:47.168339 12215 log.cc:824] T 5a13b4d29ea54a8aa8ebde8145657b25 P d8dae717db5344d380f882ece85da754: Log is configured to *not* fsync() on all Append() calls
I20260929 20:48:47.168396 12216 tablet_bootstrap.cc:654] T 5a13b4d29ea54a8aa8ebde8145657b25 P 07623714add144c6811dc479af030fad: Neither blocks nor log segments found. Creating new log.
I20260929 20:48:47.168447 12214 tablet_bootstrap.cc:492] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945: No bootstrap required, opened a new log
I20260929 20:48:47.168700 12214 ts_tablet_manager.cc:1397] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 20:48:47.169487 12216 log.cc:824] T 5a13b4d29ea54a8aa8ebde8145657b25 P 07623714add144c6811dc479af030fad: Log is configured to *not* fsync() on all Append() calls
I20260929 20:48:47.171244 12215 tablet_bootstrap.cc:492] T 5a13b4d29ea54a8aa8ebde8145657b25 P d8dae717db5344d380f882ece85da754: No bootstrap required, opened a new log
I20260929 20:48:47.171502 12215 ts_tablet_manager.cc:1397] T 5a13b4d29ea54a8aa8ebde8145657b25 P d8dae717db5344d380f882ece85da754: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 20:48:47.172443 12216 tablet_bootstrap.cc:492] T 5a13b4d29ea54a8aa8ebde8145657b25 P 07623714add144c6811dc479af030fad: No bootstrap required, opened a new log
I20260929 20:48:47.172689 12216 ts_tablet_manager.cc:1397] T 5a13b4d29ea54a8aa8ebde8145657b25 P 07623714add144c6811dc479af030fad: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 20:48:47.176378 12214 raft_consensus.cc:348] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dae717db5344d380f882ece85da754" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37211 } } peers { permanent_uuid: "df07fcd6ccb940798a6d96677c4a9945" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44827 } } peers { permanent_uuid: "07623714add144c6811dc479af030fad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35965 } }
I20260929 20:48:47.176726 12214 raft_consensus.cc:374] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:48:47.176819 12214 raft_consensus.cc:729] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df07fcd6ccb940798a6d96677c4a9945, State: Initialized, Role: FOLLOWER
I20260929 20:48:47.177404 12214 consensus_queue.cc:260] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dae717db5344d380f882ece85da754" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37211 } } peers { permanent_uuid: "df07fcd6ccb940798a6d96677c4a9945" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44827 } } peers { permanent_uuid: "07623714add144c6811dc479af030fad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35965 } }
I20260929 20:48:47.179090 12214 ts_tablet_manager.cc:1428] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260929 20:48:47.182313 12215 raft_consensus.cc:348] T 5a13b4d29ea54a8aa8ebde8145657b25 P d8dae717db5344d380f882ece85da754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dae717db5344d380f882ece85da754" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37211 } } peers { permanent_uuid: "df07fcd6ccb940798a6d96677c4a9945" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44827 } } peers { permanent_uuid: "07623714add144c6811dc479af030fad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35965 } }
I20260929 20:48:47.182739 12215 raft_consensus.cc:374] T 5a13b4d29ea54a8aa8ebde8145657b25 P d8dae717db5344d380f882ece85da754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:48:47.182833 12215 raft_consensus.cc:729] T 5a13b4d29ea54a8aa8ebde8145657b25 P d8dae717db5344d380f882ece85da754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8dae717db5344d380f882ece85da754, State: Initialized, Role: FOLLOWER
I20260929 20:48:47.182981 12216 raft_consensus.cc:348] T 5a13b4d29ea54a8aa8ebde8145657b25 P 07623714add144c6811dc479af030fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dae717db5344d380f882ece85da754" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37211 } } peers { permanent_uuid: "df07fcd6ccb940798a6d96677c4a9945" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44827 } } peers { permanent_uuid: "07623714add144c6811dc479af030fad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35965 } }
I20260929 20:48:47.183439 12216 raft_consensus.cc:374] T 5a13b4d29ea54a8aa8ebde8145657b25 P 07623714add144c6811dc479af030fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:48:47.183534 12216 raft_consensus.cc:729] T 5a13b4d29ea54a8aa8ebde8145657b25 P 07623714add144c6811dc479af030fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07623714add144c6811dc479af030fad, State: Initialized, Role: FOLLOWER
I20260929 20:48:47.183456 12215 consensus_queue.cc:260] T 5a13b4d29ea54a8aa8ebde8145657b25 P d8dae717db5344d380f882ece85da754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dae717db5344d380f882ece85da754" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37211 } } peers { permanent_uuid: "df07fcd6ccb940798a6d96677c4a9945" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44827 } } peers { permanent_uuid: "07623714add144c6811dc479af030fad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35965 } }
I20260929 20:48:47.184084 12216 consensus_queue.cc:260] T 5a13b4d29ea54a8aa8ebde8145657b25 P 07623714add144c6811dc479af030fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dae717db5344d380f882ece85da754" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37211 } } peers { permanent_uuid: "df07fcd6ccb940798a6d96677c4a9945" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44827 } } peers { permanent_uuid: "07623714add144c6811dc479af030fad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35965 } }
I20260929 20:48:47.184567 12195 heartbeater.cc:503] Master 127.0.97.62:42347 was elected leader, sending a full tablet report...
I20260929 20:48:47.185254 12215 ts_tablet_manager.cc:1428] T 5a13b4d29ea54a8aa8ebde8145657b25 P d8dae717db5344d380f882ece85da754: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 20:48:47.185678 12061 heartbeater.cc:503] Master 127.0.97.62:42347 was elected leader, sending a full tablet report...
I20260929 20:48:47.185679 12216 ts_tablet_manager.cc:1428] T 5a13b4d29ea54a8aa8ebde8145657b25 P 07623714add144c6811dc479af030fad: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260929 20:48:47.238240 12062 tablet.cc:2369] T 5a13b4d29ea54a8aa8ebde8145657b25 P 07623714add144c6811dc479af030fad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:48:47.280874 11919 tablet.cc:2369] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:48:47.303222 12196 tablet.cc:2369] T 5a13b4d29ea54a8aa8ebde8145657b25 P d8dae717db5344d380f882ece85da754: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:48:47.588792 12220 raft_consensus.cc:482] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:48:47.589148 12220 raft_consensus.cc:504] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dae717db5344d380f882ece85da754" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37211 } } peers { permanent_uuid: "df07fcd6ccb940798a6d96677c4a9945" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44827 } } peers { permanent_uuid: "07623714add144c6811dc479af030fad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35965 } }
I20260929 20:48:47.590823 12220 leader_election.cc:288] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8dae717db5344d380f882ece85da754 (127.0.97.3:37211), 07623714add144c6811dc479af030fad (127.0.97.2:35965)
I20260929 20:48:47.598110 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:47.591248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54185 (local address 127.0.97.3:37211)
0929 20:48:47.591442 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:47.591448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:47.592274 (+ 826us) server_negotiation.cc:407] Connection header received
0929 20:48:47.592434 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:47.592440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:47.592515 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:47.592601 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:48:47.593165 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.593976 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.596343 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.597138 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.597232 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:47.597311 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:47.597687 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 20:48:47.597813 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 20:48:47.598217 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:47.591295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37211 (local address 127.0.97.1:54185)
0929 20:48:47.592161 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:47.592183 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:48:47.592310 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:48:47.592618 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:48:47.592623 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:48:47.592645 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:48:47.593030 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:47.593040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.594095 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.594099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:47.596189 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:47.596201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.597360 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.597367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:47.597540 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:47.597563 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:48:47.597644 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 20:48:47.597778 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":23,"thread_start_us":563,"threads_started":1}
I20260929 20:48:47.598297 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:47.591385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40653 (local address 127.0.97.2:35965)
0929 20:48:47.591586 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:47.591591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:47.592432 (+ 841us) server_negotiation.cc:407] Connection header received
0929 20:48:47.592560 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:47.592565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:47.592648 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:47.592741 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:48:47.593310 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.594011 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.596493 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.597251 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.597373 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:47.597460 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:47.597798 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 20:48:47.597919 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:48:47.598315 12224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:47.591340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35965 (local address 127.0.97.1:40653)
0929 20:48:47.592314 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:47.592330 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:48:47.592454 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:48:47.592766 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:48:47.592769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:48:47.592800 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:48:47.593147 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:47.593158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.594133 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.594137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:47.596349 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:48:47.596362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.597482 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.597487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:48:47.597648 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:47.597674 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 20:48:47.597754 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 20:48:47.597876 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":24,"thread_start_us":531,"threads_started":1}
I20260929 20:48:47.599239 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a13b4d29ea54a8aa8ebde8145657b25" candidate_uuid: "df07fcd6ccb940798a6d96677c4a9945" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8dae717db5344d380f882ece85da754" is_pre_election: true
I20260929 20:48:47.599305 12016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a13b4d29ea54a8aa8ebde8145657b25" candidate_uuid: "df07fcd6ccb940798a6d96677c4a9945" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07623714add144c6811dc479af030fad" is_pre_election: true
I20260929 20:48:47.599671 12150 raft_consensus.cc:2454] T 5a13b4d29ea54a8aa8ebde8145657b25 P d8dae717db5344d380f882ece85da754 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df07fcd6ccb940798a6d96677c4a9945 in term 0.
I20260929 20:48:47.599766 12016 raft_consensus.cc:2454] T 5a13b4d29ea54a8aa8ebde8145657b25 P 07623714add144c6811dc479af030fad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df07fcd6ccb940798a6d96677c4a9945 in term 0.
I20260929 20:48:47.600288 11809 leader_election.cc:302] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07623714add144c6811dc479af030fad, d8dae717db5344d380f882ece85da754, df07fcd6ccb940798a6d96677c4a9945; no voters:
I20260929 20:48:47.600672 12220 raft_consensus.cc:2781] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:48:47.600788 12220 raft_consensus.cc:482] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:48:47.600858 12220 raft_consensus.cc:3037] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:48:47.603116 12220 raft_consensus.cc:504] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dae717db5344d380f882ece85da754" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37211 } } peers { permanent_uuid: "df07fcd6ccb940798a6d96677c4a9945" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44827 } } peers { permanent_uuid: "07623714add144c6811dc479af030fad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35965 } }
I20260929 20:48:47.603719 12220 leader_election.cc:288] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945 [CANDIDATE]: Term 1 election: Requested vote from peers d8dae717db5344d380f882ece85da754 (127.0.97.3:37211), 07623714add144c6811dc479af030fad (127.0.97.2:35965)
I20260929 20:48:47.604166 12150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a13b4d29ea54a8aa8ebde8145657b25" candidate_uuid: "df07fcd6ccb940798a6d96677c4a9945" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8dae717db5344d380f882ece85da754"
I20260929 20:48:47.604207 12016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a13b4d29ea54a8aa8ebde8145657b25" candidate_uuid: "df07fcd6ccb940798a6d96677c4a9945" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07623714add144c6811dc479af030fad"
I20260929 20:48:47.604343 12150 raft_consensus.cc:3037] T 5a13b4d29ea54a8aa8ebde8145657b25 P d8dae717db5344d380f882ece85da754 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:48:47.604377 12016 raft_consensus.cc:3037] T 5a13b4d29ea54a8aa8ebde8145657b25 P 07623714add144c6811dc479af030fad [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:48:47.606431 12016 raft_consensus.cc:2454] T 5a13b4d29ea54a8aa8ebde8145657b25 P 07623714add144c6811dc479af030fad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df07fcd6ccb940798a6d96677c4a9945 in term 1.
I20260929 20:48:47.606431 12150 raft_consensus.cc:2454] T 5a13b4d29ea54a8aa8ebde8145657b25 P d8dae717db5344d380f882ece85da754 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df07fcd6ccb940798a6d96677c4a9945 in term 1.
I20260929 20:48:47.606801 11809 leader_election.cc:302] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8dae717db5344d380f882ece85da754, df07fcd6ccb940798a6d96677c4a9945; no voters:
I20260929 20:48:47.607120 12220 raft_consensus.cc:2781] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:48:47.607539 12220 raft_consensus.cc:686] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945 [term 1 LEADER]: Becoming Leader. State: Replica: df07fcd6ccb940798a6d96677c4a9945, State: Running, Role: LEADER
I20260929 20:48:47.608027 12220 consensus_queue.cc:237] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dae717db5344d380f882ece85da754" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37211 } } peers { permanent_uuid: "df07fcd6ccb940798a6d96677c4a9945" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44827 } } peers { permanent_uuid: "07623714add144c6811dc479af030fad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35965 } }
I20260929 20:48:47.612258 11704 catalog_manager.cc:5495] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945 reported cstate change: term changed from 0 to 1, leader changed from <none> to df07fcd6ccb940798a6d96677c4a9945 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "df07fcd6ccb940798a6d96677c4a9945" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dae717db5344d380f882ece85da754" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df07fcd6ccb940798a6d96677c4a9945" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07623714add144c6811dc479af030fad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35965 } health_report { overall_health: UNKNOWN } } }
I20260929 20:48:47.691802 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:47.686372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44124 (local address 127.0.97.1:44827)
0929 20:48:47.687197 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:47.687201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:47.687215 (+ 14us) server_negotiation.cc:407] Connection header received
0929 20:48:47.687296 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:47.687299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:47.687380 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:47.687501 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:48:47.687904 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.688568 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.689911 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.690311 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.690399 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:47.690803 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:48:47.691348 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:48:47.691449 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:47.691524 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 20:48:47.691585 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":636,"threads_started":1}
I20260929 20:48:47.699397 12016 raft_consensus.cc:1261] T 5a13b4d29ea54a8aa8ebde8145657b25 P 07623714add144c6811dc479af030fad [term 1 FOLLOWER]: Refusing update from remote peer df07fcd6ccb940798a6d96677c4a9945: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:48:47.699402 12150 raft_consensus.cc:1261] T 5a13b4d29ea54a8aa8ebde8145657b25 P d8dae717db5344d380f882ece85da754 [term 1 FOLLOWER]: Refusing update from remote peer df07fcd6ccb940798a6d96677c4a9945: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:48:47.700304 12225 consensus_queue.cc:1035] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8dae717db5344d380f882ece85da754" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:48:47.700606 12220 consensus_queue.cc:1035] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07623714add144c6811dc479af030fad" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:48:47.712666 12230 mvcc.cc:205] Tried to move back new op lower bound from 7334768343847333888 to 7334768343486070784. Current Snapshot: MvccSnapshot[applied={T|T < 7334768343847333888}]
I20260929 20:48:47.714601 12234 mvcc.cc:205] Tried to move back new op lower bound from 7334768343847333888 to 7334768343486070784. Current Snapshot: MvccSnapshot[applied={T|T < 7334768343847333888}]
I20260929 20:48:47.725553 11704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:48:47.725791 11704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:48:47.729566 11704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eefce229379c4076b3082745c55118a3: Sending DeleteTablet for 3 replicas of tablet 5a13b4d29ea54a8aa8ebde8145657b25
I20260929 20:48:47.730613 12130 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a13b4d29ea54a8aa8ebde8145657b25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:48:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58086
I20260929 20:48:47.730830 11996 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a13b4d29ea54a8aa8ebde8145657b25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:48:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39410
I20260929 20:48:47.731081 11853 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a13b4d29ea54a8aa8ebde8145657b25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:48:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44116
I20260929 20:48:47.732856 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 11785
I20260929 20:48:47.739110 12241 tablet_replica.cc:330] T 5a13b4d29ea54a8aa8ebde8145657b25 P d8dae717db5344d380f882ece85da754: stopping tablet replica
I20260929 20:48:47.739598 12241 raft_consensus.cc:2229] T 5a13b4d29ea54a8aa8ebde8145657b25 P d8dae717db5344d380f882ece85da754 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:48:47.740092 12241 raft_consensus.cc:2258] T 5a13b4d29ea54a8aa8ebde8145657b25 P d8dae717db5344d380f882ece85da754 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:48:47.741966 12241 ts_tablet_manager.cc:1905] T 5a13b4d29ea54a8aa8ebde8145657b25 P d8dae717db5344d380f882ece85da754: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 20:48:47 dist-test-slave-blv7 krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:48:47.750329 12241 ts_tablet_manager.cc:1918] T 5a13b4d29ea54a8aa8ebde8145657b25 P d8dae717db5344d380f882ece85da754: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:48:47.750480 12241 log.cc:1196] T 5a13b4d29ea54a8aa8ebde8145657b25 P d8dae717db5344d380f882ece85da754: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/5a13b4d29ea54a8aa8ebde8145657b25
I20260929 20:48:47.750952 12241 ts_tablet_manager.cc:1939] T 5a13b4d29ea54a8aa8ebde8145657b25 P d8dae717db5344d380f882ece85da754: Deleting consensus metadata
I20260929 20:48:47.751526 12242 tablet_replica.cc:330] T 5a13b4d29ea54a8aa8ebde8145657b25 P 07623714add144c6811dc479af030fad: stopping tablet replica
I20260929 20:48:47.751988 12242 raft_consensus.cc:2229] T 5a13b4d29ea54a8aa8ebde8145657b25 P 07623714add144c6811dc479af030fad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:48:47.752048 12243 tablet_replica.cc:330] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945: stopping tablet replica
I20260929 20:48:47.752458 12242 raft_consensus.cc:2258] T 5a13b4d29ea54a8aa8ebde8145657b25 P 07623714add144c6811dc479af030fad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:48:47.752622 12243 raft_consensus.cc:2229] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:48:47.753347 12243 raft_consensus.cc:2258] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:48:47.754269 12242 ts_tablet_manager.cc:1905] T 5a13b4d29ea54a8aa8ebde8145657b25 P 07623714add144c6811dc479af030fad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:48:47.755260 12243 ts_tablet_manager.cc:1905] T 5a13b4d29ea54a8aa8ebde8145657b25 P df07fcd6ccb940798a6d96677c4a9945: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:48:47.756944 11692 catalog_manager.cc:4841] TS d8dae717db5344d380f882ece85da754 (127.0.97.3:37211): tablet 5a13b4d29ea54a8aa8ebde8145657b25 (table test-table [id=da482bbe2d6b46d3b3afe7ee79f1dcd8]) successfully deleted
I20260929 20:48:47.759123 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:47.733894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44136 (local address 127.0.97.1:44827)
0929 20:48:47.734059 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:47.734065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:47.738400 (+ 4335us) server_negotiation.cc:407] Connection header received
0929 20:48:47.738616 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:47.738622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:47.738702 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:47.738798 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:48:47.740436 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.741688 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.743789 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.744465 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.744582 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:47.750232 (+ 5650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:48:47.750250 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:48:47.750270 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:48:47.750324 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:48:47.754349 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:47.755214 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:47.755222 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:47.755226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:47.755404 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:47.755893 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:47.755898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:47.755901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:47.756311 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:48:47.757265 (+ 954us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:47.758489 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0929 20:48:47.758733 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 20:48:47.760006 11913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:48:47.761736 12242 ts_tablet_manager.cc:1918] T 5a13b4d29ea54a8aa8ebde8145657b25 P 07623714add144c6811dc479af030fad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:48:47.761947 12242 log.cc:1196] T 5a13b4d29ea54a8aa8ebde8145657b25 P 07623714add144c6811dc479af030fad: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/5a13b4d29ea54a8aa8ebde8145657b25
I20260929 20:48:47.762509 12242 ts_tablet_manager.cc:1939] T 5a13b4d29ea54a8aa8ebde8145657b25 P 07623714add144c6811dc479af030fad: Deleting consensus metadata
I20260929 20:48:47.764305 11689 catalog_manager.cc:4841] TS 07623714add144c6811dc479af030fad (127.0.97.2:35965): tablet 5a13b4d29ea54a8aa8ebde8145657b25 (table test-table [id=da482bbe2d6b46d3b3afe7ee79f1dcd8]) successfully deleted
I20260929 20:48:47.854768 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 11785
W20260929 20:48:47.868628 11689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44827 (error 108)
I20260929 20:48:47.869338 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 11924
I20260929 20:48:47.870744 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:47.869931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44827 (local address 127.0.0.1:44148)
0929 20:48:47.870488 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:47.870580 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":325,"threads_started":1}
W20260929 20:48:47.871093 11689 catalog_manager.cc:4584] TS df07fcd6ccb940798a6d96677c4a9945 (127.0.97.1:44827): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a13b4d29ea54a8aa8ebde8145657b25: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44827: connect: Connection refused (error 111)
Sep 29 20:48:47 dist-test-slave-blv7 krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:48:47.884006 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:47.870084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39424 (local address 127.0.97.2:35965)
0929 20:48:47.870228 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:47.870232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:47.870406 (+ 174us) server_negotiation.cc:407] Connection header received
0929 20:48:47.870618 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:47.870623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:47.870699 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:47.870786 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:48:47.872215 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.872909 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.874180 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.874593 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.874699 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:47.878637 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:48:47.878653 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:48:47.878670 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:48:47.878709 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:48:47.881303 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:47.881988 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:47.881996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:47.881998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:47.882127 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:47.882459 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:47.882464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:47.882466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:47.882813 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:48:47.883098 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:47.883491 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 20:48:47.883698 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 20:48:47.884727 12056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:48:47.924962 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:47.924431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44827 (local address 127.0.0.1:44154)
0929 20:48:47.924673 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:47.924795 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 20:48:47.926033 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:47.925643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44827 (local address 127.0.0.1:44162)
0929 20:48:47.925781 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:47.925878 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 20:48:47.971628 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 11924
I20260929 20:48:47.984310 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 12067
I20260929 20:48:47.986142 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:47.985682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44827 (local address 127.0.0.1:44172)
0929 20:48:47.985872 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:47.985973 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 20:48:47.987210 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:47.986750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44827 (local address 127.0.0.1:44188)
0929 20:48:47.986893 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:47.986977 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
Sep 29 20:48:47 dist-test-slave-blv7 krb5kdc[11229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:48:48.000128 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:47.984986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58094 (local address 127.0.97.3:37211)
0929 20:48:47.985170 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 20:48:47.985176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:48:47.985197 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:48:47.985387 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:48:47.985392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:48:47.985487 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:48:47.985597 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:48:47.987198 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.987996 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.989226 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:48:47.989620 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:48:47.989751 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:48:47.993858 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:48:47.993873 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:48:47.993891 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:48:47.993931 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:48:47.996742 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:47.997683 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:47.997693 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:47.997697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:47.997906 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:48:47.998384 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:48:47.998390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:48:47.998393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:48:47.998911 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:48:47.999347 (+ 436us) server_negotiation.cc:1035] Waiting for connection context
0929 20:48:47.999678 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 20:48:47.999843 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:48:48.000950 12190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:48:48.067723 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:48.067277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44827 (local address 127.0.0.1:44202)
0929 20:48:48.067478 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:48.067559 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 20:48:48.068800 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:48:48.068426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44827 (local address 127.0.0.1:44206)
0929 20:48:48.068601 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 20:48:48.068668 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 20:48:48.102474 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 12067
I20260929 20:48:48.115165 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 11673
I20260929 20:48:48.116308 11735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:48:48.226083 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 11673
I20260929 20:48:48.242545 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 20:48:48.577880 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 20:48:48.578100 388 mini_ranger.cc:66] Stopping Ranger...
I20260929 20:48:48.923448 388 mini_ranger.cc:68] Stopped Ranger
2026-09-29 20:48:48.923 UTC [11259] LOG: received smart shutdown request
2026-09-29 20:48:48.927 UTC [11259] LOG: background worker "logical replication launcher" (PID 11268) exited with exit code 1
2026-09-29 20:48:48.927 UTC [11263] LOG: shutting down
2026-09-29 20:48:49.095 UTC [11259] LOG: database system is shut down
2026-09-29T20:48:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46795 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:48:49 dist-test-slave-blv7 krb5kdc[12259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:48:49 dist-test-slave-blv7 krb5kdc[12259](info): set up 2 sockets
Sep 29 20:48:49 dist-test-slave-blv7 krb5kdc[12259](info): commencing operation
krb5kdc: starting...
W20260929 20:48:50.211272 388 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:48:50 dist-test-slave-blv7 krb5kdc[12259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:48:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:48:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260929 20:48:50.518759 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskf3T6XB/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/postgres -l logfile start
2026-09-29 20:48:52.919 UTC [12289] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 20:48:52.919 UTC [12289] LOG: listening on IPv4 address "127.0.97.20", port 35091
2026-09-29 20:48:52.923 UTC [12289] LOG: listening on Unix socket "/tmp/.s.PGSQL.35091"
2026-09-29 20:48:52.945 UTC [12292] LOG: database system was shut down at 2026-09-29 20:48:51 UTC
2026-09-29 20:48:52.948 UTC [12289] LOG: database system is ready to accept connections
I20260929 20:48:53.838701 388 mini_postgres.cc:96] Postgres bound to 35091
2026-09-29 20:48:53.853 UTC [12300] FATAL: database "slave" does not exist
127.0.97.20:35091 - accepting connections
I20260929 20:48:53.854409 388 mini_ranger.cc:161] Starting Ranger...
I20260929 20:48:53.877696 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 20:48:54.350057 388 mini_ranger.cc:87] Created ranger Postgres database
I20260929 20:48:54.350145 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ranger-admin
2026-09-29 20:48:54,605 [I] DB FLAVOR :POSTGRES
2026-09-29 20:48:54,605 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 20:48:54,605 [I] Checking connection..
2026-09-29 20:48:54,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 20:48:54,862 [I] Checking connection passed.
2026-09-29 20:48:54,862 [I] --------- Verifying version history table ---------
2026-09-29 20:48:54,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:48:55,117 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 20:48:55,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:48:55,362 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 20:48:55,363 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 20:48:55,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 20:48:55.590 UTC [12364] WARNING: there is no transaction in progress
2026-09-29 20:48:55,596 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 20:48:55,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:48:55,853 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 20:48:55,853 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 20:48:55,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 20:48:56,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 20:48:56,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-blv7.c.gcp-upstream.internal','N') ;"
2026-09-29 20:48:56,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 20:48:56,817 [I] Table x_portal_user does not exist in database ranger
2026-09-29 20:48:56,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 20:48:57,062 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 20:48:57,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 20:48:57,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 20:48:57,542 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 20:48:57,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 20:48:58.490 UTC [12472] WARNING: there is no transaction in progress
2026-09-29 20:48:58.505 UTC [12472] WARNING: there is no transaction in progress
2026-09-29 20:48:58.520 UTC [12472] WARNING: there is no transaction in progress
2026-09-29 20:48:58.535 UTC [12472] WARNING: there is no transaction in progress
2026-09-29 20:48:58.551 UTC [12472] WARNING: there is no transaction in progress
2026-09-29 20:48:58.567 UTC [12472] WARNING: there is no transaction in progress
2026-09-29 20:48:58.619 UTC [12472] WARNING: there is no transaction in progress
2026-09-29 20:48:58.625 UTC [12472] WARNING: there is no transaction in progress
2026-09-29 20:48:58.640 UTC [12472] WARNING: there is no transaction in progress
2026-09-29 20:48:58.650 UTC [12472] WARNING: there is no transaction in progress
2026-09-29 20:48:58.660 UTC [12472] WARNING: there is no transaction in progress
2026-09-29 20:48:58.674 UTC [12472] WARNING: there is no transaction in progress
2026-09-29 20:48:58.683 UTC [12472] WARNING: there is no transaction in progress
2026-09-29 20:48:58.689 UTC [12472] WARNING: there is no transaction in progress
2026-09-29 20:48:58.696 UTC [12472] WARNING: there is no transaction in progress
2026-09-29 20:48:58.702 UTC [12472] WARNING: there is no transaction in progress
2026-09-29 20:48:59,309 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 20:48:59,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 20:48:59,541 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 20:48:59,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 20:48:59,781 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 20:48:59,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 20:49:00,012 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 20:49:00,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 20:49:00,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 20:49:00,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-blv7.c.gcp-upstream.internal';"
2026-09-29 20:49:00,742 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 20:49:00,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 20:49:00,993 [I] DB_PATCHES have already been applied
I20260929 20:49:00.998741 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskf3T6XB/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 20:49:00.998828 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260929 20:49:01.011595 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:37525
Sep 29, 2026 8:49:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 8:49:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 8:49:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 8:49:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 8:49:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 8:49:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 8:49:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 20:49:02.019721 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.165s user 0.001s sys 0.034s
Sep 29, 2026 8:49:22 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 8:49:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 8:49:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 8:49:22 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 8:49:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 8:49:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 8:49:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 8:49:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 8:49:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 8:49:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 8:49:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 8:49:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 8:49:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 8:49:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 8:49:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 8:49:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 8:49:22 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 8:49:23 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 20:49:23.701599 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 20:49:23 dist-test-slave-blv7 krb5kdc[12259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714963, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 20:49:23.865989 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 20:49:23.884123 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 20:49:24.732600 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 20:49:24.732702 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ranger-kms
I20260929 20:49:24.732736 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:35091
2026-09-29 20:49:25,050 [I] DB FLAVOR :POSTGRES
2026-09-29 20:49:25,050 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 20:49:25,050 [I] Checking connection
2026-09-29 20:49:25,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 20:49:25,276 [I] connection success
2026-09-29 20:49:25,276 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 20:49:25,276 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 20:49:25,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 20:49:25,507 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 20:49:25,507 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 20:49:25,507 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 20:49:25,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:35091/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 20:49:25,752 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 20:49:26.187218 388 mini_ranger_kms.cc:310] Created kms service
I20260929 20:49:26.401372 388 mini_ranger_kms.cc:326] Created kudu user
I20260929 20:49:26.448820 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 20:49:26 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714963, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:49:26.820596 388 mini_ranger_kms.cc:384] Added ranger policy
I20260929 20:49:26.820888 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskf3T6XB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskf3T6XB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 20:49:26.820953 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260929 20:49:26.834071 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12652
I20260929 20:49:26.834189 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:46161
Sep 29, 2026 8:49:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 8:49:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 8:49:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 8:49:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 8:49:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 8:49:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 20:49:28.894327 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.028s user 0.064s sys 0.268s
I20260929 20:49:28.894562 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 20:49:28.894704 388 mini_ranger_kms.cc:401] 127.0.97.20:46161/kms/v1/keys
Sep 29 20:49:29 dist-test-slave-blv7 krb5kdc[12259](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790714969, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 20:49:29 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714969, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 20:49:29 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714969, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 20:49:31 dist-test-slave-blv7 krb5kdc[12259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:49:32.011185 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38013
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38013
--ranger_config_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46161/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:49:32.340497 12703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:32.340757 12703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:32.340883 12703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:32.347839 12703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:49:32.347931 12703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:32.347999 12703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:32.348071 12703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:49:32.348131 12703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:49:32.357625 12703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:46161/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38013
--ranger_config_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:32.358774 12703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:32.360051 12703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:32.370904 12708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:32.370939 12709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:32.370812 12711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:32.371917 12703 server_base.cc:1034] running on GCE node
I20260929 20:49:32.372649 12703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:32.373939 12703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:32.375099 12703 hybrid_clock.cc:648] HybridClock initialized: now 1790714972375065 us; error 27 us; skew 500 ppm
I20260929 20:49:32.375419 12703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:32 dist-test-slave-blv7 krb5kdc[12259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:32.381806 12703 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:49:32.383205 12703 webserver.cc:460] Webserver started at http://127.0.97.62:42571/ using document root <none> and password file <none>
I20260929 20:49:32.383801 12703 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:32.383903 12703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:49:32.384169 12703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:49:32 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:49:32.437228 12703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "31805551e99449429984026ee29db309"
format_stamp: "Formatted at 2026-09-29 20:49:32 on dist-test-slave-blv7"
server_key: "bd6cf147286f40a2f920606589033f4d"
server_key_iv: "6351dae8e8defbc353997d344c6418f6"
server_key_version: "kuduclusterkey@0"
I20260929 20:49:32.438036 12703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "31805551e99449429984026ee29db309"
format_stamp: "Formatted at 2026-09-29 20:49:32 on dist-test-slave-blv7"
server_key: "bd6cf147286f40a2f920606589033f4d"
server_key_iv: "6351dae8e8defbc353997d344c6418f6"
server_key_version: "kuduclusterkey@0"
I20260929 20:49:32.443795 12703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:49:32.464790 12718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:32.466322 12703 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:49:32.466488 12703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "31805551e99449429984026ee29db309"
format_stamp: "Formatted at 2026-09-29 20:49:32 on dist-test-slave-blv7"
server_key: "bd6cf147286f40a2f920606589033f4d"
server_key_iv: "6351dae8e8defbc353997d344c6418f6"
server_key_version: "kuduclusterkey@0"
I20260929 20:49:32.466706 12703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:32.496235 12703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:32.500145 12703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:32.500429 12703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:32.517621 12703 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38013
I20260929 20:49:32.517642 12769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38013 every 8 connection(s)
I20260929 20:49:32.518916 12703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:49:32.522502 12770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:49:32.524735 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 12703
I20260929 20:49:32.524919 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:49:32.525517 388 external_mini_cluster.cc:1428] Setting key 9746db6d02456a88d30a4a4fa3291567
I20260929 20:49:32.533625 12770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309: Bootstrap starting.
I20260929 20:49:32.537087 12770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309: Neither blocks nor log segments found. Creating new log.
I20260929 20:49:32.538295 12770 log.cc:824] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:49:32 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:49:32.541591 12770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309: No bootstrap required, opened a new log
I20260929 20:49:32.549525 12773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:32.527445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57496 (local address 127.0.97.62:38013)
0929 20:49:32.528176 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:32.528187 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:32.528228 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:49:32.529848 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:32.529870 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:32.530305 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:32.530643 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:32.532257 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:32.533528 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:32.535287 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:32.535931 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:32.536062 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:32.540762 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:32.540773 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:32.540792 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:32.540834 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:32.544692 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:32.545611 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:32.545619 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:32.545627 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:32.545834 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:32.546269 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:32.546276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:32.546280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:32.546849 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:32.547189 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:32.547847 (+ 658us) server_negotiation.cc:299] Negotiation successful
0929 20:49:32.548258 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":341,"threads_started":1}
I20260929 20:49:32.551935 12770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31805551e99449429984026ee29db309" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38013 } }
I20260929 20:49:32.552364 12770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:49:32.552476 12770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31805551e99449429984026ee29db309, State: Initialized, Role: FOLLOWER
I20260929 20:49:32.553148 12770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31805551e99449429984026ee29db309" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38013 } }
I20260929 20:49:32.553435 12770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:49:32.553615 12770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:49:32.553788 12770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:49:32.556384 12770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31805551e99449429984026ee29db309" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38013 } }
I20260929 20:49:32.556906 12770 leader_election.cc:302] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31805551e99449429984026ee29db309; no voters:
I20260929 20:49:32.557472 12770 leader_election.cc:288] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:49:32.557561 12775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:49:32.558125 12775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [term 1 LEADER]: Becoming Leader. State: Replica: 31805551e99449429984026ee29db309, State: Running, Role: LEADER
I20260929 20:49:32.558681 12775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31805551e99449429984026ee29db309" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38013 } }
I20260929 20:49:32.559602 12770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:49:32.561269 12776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31805551e99449429984026ee29db309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31805551e99449429984026ee29db309" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38013 } } }
I20260929 20:49:32.561636 12776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [sys.catalog]: This master's current role is: LEADER
I20260929 20:49:32.562248 12777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31805551e99449429984026ee29db309. Latest consensus state: current_term: 1 leader_uuid: "31805551e99449429984026ee29db309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31805551e99449429984026ee29db309" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38013 } } }
I20260929 20:49:32.562564 12777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [sys.catalog]: This master's current role is: LEADER
I20260929 20:49:32.563932 12781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:49:32.568192 12781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:49:32.576684 12781 catalog_manager.cc:1347] Generated new cluster ID: 2bf6635840734132af94c42d5f8f3b9c
I20260929 20:49:32.576783 12781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:49:32.577096 12770 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 20:49:32.603188 12781 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:49:32.605302 12781 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:49:32.631605 12781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309: Generated new TSK 0
I20260929 20:49:32.632514 12781 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:49:33.378489 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38013
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:49:33 dist-test-slave-blv7 krb5kdc[12259](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790714973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 20:49:33.813215 12815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:33.813681 12815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:33.813884 12815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:33.825191 12815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:33.825392 12815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:33.825618 12815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:49:33.841022 12815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:46161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:33.842950 12815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:33.844802 12815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:33.863003 12825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:33.864032 12826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:33.869045 12828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:33.869392 12815 server_base.cc:1034] running on GCE node
I20260929 20:49:33.870350 12815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:33.871682 12815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:33.872988 12815 hybrid_clock.cc:648] HybridClock initialized: now 1790714973872912 us; error 55 us; skew 500 ppm
I20260929 20:49:33.873373 12815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:33 dist-test-slave-blv7 krb5kdc[12259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:33.877630 12815 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:49:33.879616 12815 webserver.cc:460] Webserver started at http://127.0.97.1:46487/ using document root <none> and password file <none>
I20260929 20:49:33.880568 12815 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:33.880698 12815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:49:33.881039 12815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:49:33 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:49:33.920184 12815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "d3151417daca4f54860975b65dbb83c4"
format_stamp: "Formatted at 2026-09-29 20:49:33 on dist-test-slave-blv7"
server_key: "fbf3acbb308aa62106e5be36671f155d"
server_key_iv: "b1efacd110d0a21d9328dcd5a135ee94"
server_key_version: "kuduclusterkey@0"
I20260929 20:49:33.921372 12815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d3151417daca4f54860975b65dbb83c4"
format_stamp: "Formatted at 2026-09-29 20:49:33 on dist-test-slave-blv7"
server_key: "fbf3acbb308aa62106e5be36671f155d"
server_key_iv: "b1efacd110d0a21d9328dcd5a135ee94"
server_key_version: "kuduclusterkey@0"
I20260929 20:49:33.927609 12815 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260929 20:49:33.945868 12835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:33.947979 12815 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:49:33.948268 12815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "d3151417daca4f54860975b65dbb83c4"
format_stamp: "Formatted at 2026-09-29 20:49:33 on dist-test-slave-blv7"
server_key: "fbf3acbb308aa62106e5be36671f155d"
server_key_iv: "b1efacd110d0a21d9328dcd5a135ee94"
server_key_version: "kuduclusterkey@0"
I20260929 20:49:33.948640 12815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:34.044945 12815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:34.050195 12815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:34.050583 12815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:34.058277 12815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:49:34.075462 12815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:49:34.075657 12815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 20:49:34.075870 12815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:49:34.076004 12815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:49:34 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:49:34.317664 12815 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41199
I20260929 20:49:34.317708 12947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41199 every 8 connection(s)
I20260929 20:49:34.319461 12815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:49:34.329282 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 12815
I20260929 20:49:34.329468 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:49:34.330207 388 external_mini_cluster.cc:1428] Setting key d1d986911aa08c0b2ccf941c4d353f77
Sep 29 20:49:34 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 20:49:34.393524 12951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:34.323719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36761 (local address 127.0.97.62:38013)
0929 20:49:34.356618 (+ 32899us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:34.356624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:34.366718 (+ 10094us) server_negotiation.cc:407] Connection header received
0929 20:49:34.366796 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:34.366801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:34.366893 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:34.366999 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:34.370891 (+ 3892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:34.371992 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:34.377269 (+ 5277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:34.377867 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:34.377978 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:34.386170 (+ 8192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:34.386178 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:34.386183 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:34.386235 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:34.389736 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:34.390695 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:34.390702 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:34.390705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:34.390861 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:34.391445 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:34.391451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:34.391454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:34.391792 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:34.391920 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:34.392968 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0929 20:49:34.393183 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32747,"thread_start_us":32916,"threads_started":1}
I20260929 20:49:34.395953 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:34.323087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38013 (local address 127.0.97.1:36761)
0929 20:49:34.363799 (+ 40712us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:34.363865 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:34.366256 (+ 2391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:34.368230 (+ 1974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:34.368244 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:34.368834 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:49:34.370682 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:34.370708 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:34.372228 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:34.372232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:34.374117 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:34.374138 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:34.378331 (+ 4193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:34.378338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:34.378618 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:34.383217 (+ 4599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:49:34.383258 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:49:34.385847 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:49:34.389938 (+ 4091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:34.389949 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:34.389969 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:34.390548 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:34.391053 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:34.391058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:34.391062 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:34.391314 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:34.392029 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:49:34.392037 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:49:34.392493 (+ 456us) client_negotiation.cc:769] Sending connection context
0929 20:49:34.394041 (+ 1548us) client_negotiation.cc:240] Negotiation successful
0929 20:49:34.394570 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40346,"thread_start_us":43932,"threads_started":1}
I20260929 20:49:34.398434 12948 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38013
I20260929 20:49:34.398741 12948 heartbeater.cc:464] Registering TS with master...
I20260929 20:49:34.399562 12948 heartbeater.cc:511] Master 127.0.97.62:38013 requested a full tablet report, sending...
I20260929 20:49:34.401968 12735 ts_manager.cc:194] Registered new tserver with Master: d3151417daca4f54860975b65dbb83c4 (127.0.97.1:41199)
I20260929 20:49:34.403976 12735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
W20260929 20:49:34.448890 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.2: real 0.116s user 0.000s sys 0.016s
I20260929 20:49:34.449831 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38013
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:49:34 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260929 20:49:34.897800 12954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:34.898176 12954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:34.898320 12954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:34.907225 12954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:34.907334 12954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:34.907487 12954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:49:34.917205 12954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:46161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:34.918484 12954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:34.919826 12954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:34.932592 12969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:34.932614 12968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:34.933411 12954 server_base.cc:1034] running on GCE node
W20260929 20:49:34.933544 12971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:34.934132 12954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:34.935096 12954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:34.936332 12954 hybrid_clock.cc:648] HybridClock initialized: now 1790714974936283 us; error 45 us; skew 500 ppm
I20260929 20:49:34.936633 12954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:34 dist-test-slave-blv7 krb5kdc[12259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:34.939829 12954 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:49:34.941463 12954 webserver.cc:460] Webserver started at http://127.0.97.2:44419/ using document root <none> and password file <none>
I20260929 20:49:34.942299 12954 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:34.942425 12954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:49:34.942749 12954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:49:34 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:49:34.960037 12954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "522b57e621e845e6aa9f565598e39825"
format_stamp: "Formatted at 2026-09-29 20:49:34 on dist-test-slave-blv7"
server_key: "b5aebcc339adb785fe858866b9548636"
server_key_iv: "1054bd899cf2cbef5be968ea2e8dc2f2"
server_key_version: "kuduclusterkey@0"
I20260929 20:49:34.961076 12954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "522b57e621e845e6aa9f565598e39825"
format_stamp: "Formatted at 2026-09-29 20:49:34 on dist-test-slave-blv7"
server_key: "b5aebcc339adb785fe858866b9548636"
server_key_iv: "1054bd899cf2cbef5be968ea2e8dc2f2"
server_key_version: "kuduclusterkey@0"
I20260929 20:49:34.968065 12954 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260929 20:49:34.985817 12978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:34.987703 12954 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 20:49:34.987953 12954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "522b57e621e845e6aa9f565598e39825"
format_stamp: "Formatted at 2026-09-29 20:49:34 on dist-test-slave-blv7"
server_key: "b5aebcc339adb785fe858866b9548636"
server_key_iv: "1054bd899cf2cbef5be968ea2e8dc2f2"
server_key_version: "kuduclusterkey@0"
I20260929 20:49:34.988322 12954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:35.017409 12954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:35.022852 12954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:35.023411 12954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:35.024814 12954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:49:35.026647 12954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:49:35.026778 12954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 20:49:35.026947 12954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:49:35.027196 12954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:35.065564 12954 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43541
I20260929 20:49:35.065613 13090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43541 every 8 connection(s)
I20260929 20:49:35.067323 12954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:49:35.074429 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 12954
I20260929 20:49:35.074601 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:49:35.075179 388 external_mini_cluster.cc:1428] Setting key 9f8496e913879dafd4afa24c937eac1c
Sep 29 20:49:35 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:49:35.097349 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:35.070833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38013 (local address 127.0.97.2:44439)
0929 20:49:35.075665 (+ 4832us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:35.075718 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:35.077954 (+ 2236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:35.079087 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:35.079100 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:35.079685 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:49:35.080459 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:35.080483 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:35.081974 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:35.081979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:35.083980 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:35.084014 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:35.085157 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:35.085163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:35.085439 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:35.086239 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:49:35.086267 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:49:35.088871 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:49:35.092675 (+ 3804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:35.092684 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:35.092701 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:35.093262 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:35.093671 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:35.093676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:35.093680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:35.094053 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:35.094754 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:49:35.094762 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:49:35.095267 (+ 505us) client_negotiation.cc:769] Sending connection context
0929 20:49:35.095534 (+ 267us) client_negotiation.cc:240] Negotiation successful
0929 20:49:35.096035 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4533,"thread_start_us":14687,"threads_started":1}
I20260929 20:49:35.098395 13094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:35.071439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44439 (local address 127.0.97.62:38013)
0929 20:49:35.071895 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:35.071902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:35.078390 (+ 6488us) server_negotiation.cc:407] Connection header received
0929 20:49:35.078452 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:35.078458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:35.078546 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:35.078641 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:35.080644 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:35.081772 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:35.084155 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:35.084750 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:35.084841 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:35.089127 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:35.089136 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:35.089140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:35.089189 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:35.092504 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:35.093401 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:35.093407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:35.093410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:35.093556 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:35.094177 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:35.094229 (+ 52us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:35.094232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:35.094560 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:35.094690 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:35.097806 (+ 3116us) server_negotiation.cc:299] Negotiation successful
0929 20:49:35.098043 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":224,"threads_started":1}
I20260929 20:49:35.100349 13091 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38013
I20260929 20:49:35.100644 13091 heartbeater.cc:464] Registering TS with master...
I20260929 20:49:35.101449 13091 heartbeater.cc:511] Master 127.0.97.62:38013 requested a full tablet report, sending...
I20260929 20:49:35.102985 12735 ts_manager.cc:194] Registered new tserver with Master: 522b57e621e845e6aa9f565598e39825 (127.0.97.2:43541)
I20260929 20:49:35.104048 12735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44439
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:49:35.138896 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38013
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:49:35.407908 12948 heartbeater.cc:503] Master 127.0.97.62:38013 was elected leader, sending a full tablet report...
W20260929 20:49:35.546908 13097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:35.547235 13097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:35.547374 13097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:35.554464 13097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:35.554572 13097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:35.554709 13097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:49:35.564441 13097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:46161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:35.565663 13097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:35.566874 13097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:35.580945 13102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:35.581359 13103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:35.580917 13105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:35.582080 13097 server_base.cc:1034] running on GCE node
I20260929 20:49:35.582633 13097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:35.583639 13097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:35.584837 13097 hybrid_clock.cc:648] HybridClock initialized: now 1790714975584787 us; error 46 us; skew 500 ppm
I20260929 20:49:35.585119 13097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:35 dist-test-slave-blv7 krb5kdc[12259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:35.588238 13097 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:49:35.589623 13097 webserver.cc:460] Webserver started at http://127.0.97.3:43961/ using document root <none> and password file <none>
I20260929 20:49:35.590253 13097 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:35.590348 13097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:49:35.590590 13097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:49:35 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:49:35.605402 13097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "4a80a03682ef4d82abc86e912b6a7a58"
format_stamp: "Formatted at 2026-09-29 20:49:35 on dist-test-slave-blv7"
server_key: "536a791d9ff7ab1eca3c0ce928f25716"
server_key_iv: "5c9959049301889d11905bfadad08be6"
server_key_version: "kuduclusterkey@0"
I20260929 20:49:35.606155 13097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4a80a03682ef4d82abc86e912b6a7a58"
format_stamp: "Formatted at 2026-09-29 20:49:35 on dist-test-slave-blv7"
server_key: "536a791d9ff7ab1eca3c0ce928f25716"
server_key_iv: "5c9959049301889d11905bfadad08be6"
server_key_version: "kuduclusterkey@0"
I20260929 20:49:35.611989 13097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 20:49:35.627413 13112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:35.629091 13097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:49:35.629274 13097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "4a80a03682ef4d82abc86e912b6a7a58"
format_stamp: "Formatted at 2026-09-29 20:49:35 on dist-test-slave-blv7"
server_key: "536a791d9ff7ab1eca3c0ce928f25716"
server_key_iv: "5c9959049301889d11905bfadad08be6"
server_key_version: "kuduclusterkey@0"
I20260929 20:49:35.629499 13097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:35.662954 13097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:35.666390 13097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:35.666617 13097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:35.667593 13097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:49:35.668850 13097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:49:35.668934 13097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:35.669028 13097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:49:35.669070 13097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:35.698388 13097 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35395
I20260929 20:49:35.698426 13224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35395 every 8 connection(s)
I20260929 20:49:35.699697 13097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:49:35.709844 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 13097
I20260929 20:49:35.710005 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:49:35.710563 388 external_mini_cluster.cc:1428] Setting key 79405337b5dd8134e01626c302d87d3c
Sep 29 20:49:35 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:49:35.717944 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:35.702043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45963 (local address 127.0.97.62:38013)
0929 20:49:35.702502 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:35.702506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:35.703401 (+ 895us) server_negotiation.cc:407] Connection header received
0929 20:49:35.705096 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:35.705101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:35.705188 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:35.705281 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:35.706667 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:35.707415 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:35.708955 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:35.709408 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:35.709517 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:35.712901 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:35.712906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:35.712910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:35.712940 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:35.715378 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:35.716043 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:35.716047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:35.716049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:35.716164 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:35.716585 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:35.716588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:35.716591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:35.716832 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:35.716937 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:35.717543 (+ 606us) server_negotiation.cc:299] Negotiation successful
0929 20:49:35.717705 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":213,"threads_started":1}
I20260929 20:49:35.718873 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:35.702511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38013 (local address 127.0.97.3:45963)
0929 20:49:35.703250 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:35.703288 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:35.704829 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:35.705527 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:35.705534 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:35.705948 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:49:35.706503 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:35.706520 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:35.707563 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:35.707567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:35.708817 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:35.708829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:35.709620 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:35.709628 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:35.709836 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:35.710477 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:49:35.710511 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:49:35.712697 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:49:35.715492 (+ 2795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:35.715498 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:35.715509 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:35.715946 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:35.716278 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:35.716281 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:35.716284 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:35.716495 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:35.716933 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:49:35.716939 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:49:35.717228 (+ 289us) client_negotiation.cc:769] Sending connection context
0929 20:49:35.717469 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 20:49:35.717828 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":290,"threads_started":1}
I20260929 20:49:35.720265 13225 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38013
I20260929 20:49:35.720541 13225 heartbeater.cc:464] Registering TS with master...
I20260929 20:49:35.721246 13225 heartbeater.cc:511] Master 127.0.97.62:38013 requested a full tablet report, sending...
I20260929 20:49:35.722637 12735 ts_manager.cc:194] Registered new tserver with Master: 4a80a03682ef4d82abc86e912b6a7a58 (127.0.97.3:35395)
I20260929 20:49:35.723475 12735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45963
I20260929 20:49:35.725946 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:49:35.727653 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 12815
Sep 29 20:49:35 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:49:35.743162 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:35.728520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46744 (local address 127.0.97.1:41199)
0929 20:49:35.729065 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:35.729073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:35.729100 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:49:35.729171 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:35.729178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:35.729368 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:35.729501 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:35.730870 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:35.731911 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:35.733452 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:35.733991 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:35.734119 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:35.738030 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:35.738041 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:35.738054 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:35.738097 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:35.740611 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:35.741216 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:35.741223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:35.741225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:35.741357 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:35.741694 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:35.741697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:35.741699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:35.742010 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:35.742280 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:35.742566 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 20:49:35.742809 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":213,"threads_started":1}
I20260929 20:49:35.744468 12943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:49:35.825472 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 12815
I20260929 20:49:35.837877 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 12954
Sep 29 20:49:35 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:49:35.852638 13232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:35.838663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35684 (local address 127.0.97.2:43541)
0929 20:49:35.839209 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:35.839216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:35.839241 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:49:35.839299 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:35.839305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:35.839511 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:35.839632 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:35.840812 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:35.841802 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:35.843239 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:35.843824 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:35.843925 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:35.847941 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:35.847953 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:35.847968 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:35.848008 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:35.850404 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:35.850964 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:35.850971 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:35.850973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:35.851142 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:35.851436 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:35.851439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:35.851441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:35.851748 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:35.851965 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:35.852176 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 20:49:35.852366 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":267,"threads_started":1}
I20260929 20:49:35.853816 13086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:49:35.932595 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 12954
I20260929 20:49:35.944497 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 13097
Sep 29 20:49:35 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:49:35.959970 13234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:35.945362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46328 (local address 127.0.97.3:35395)
0929 20:49:35.945958 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:35.945965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:35.945994 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:49:35.946057 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:35.946063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:35.946264 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:35.946405 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:35.947846 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:35.948830 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:35.950097 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:35.950635 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:35.950742 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:35.954734 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:35.954744 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:35.954758 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:35.954787 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:35.957290 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:35.957907 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:35.957913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:35.957916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:35.958040 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:35.958377 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:35.958381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:35.958383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:35.958704 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:35.959125 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:35.959396 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 20:49:35.959641 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":297,"threads_started":1}
I20260929 20:49:35.961174 13220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:49:36.040676 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 13097
I20260929 20:49:36.052744 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 12703
I20260929 20:49:36.053900 12765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:49:36.155387 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 12703
I20260929 20:49:36.173949 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38013
--webserver_interface=127.0.97.62
--webserver_port=42571
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38013
--ranger_config_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46161/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46161/kms with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:49:36.482235 13237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:36.482525 13237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:36.482656 13237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:36.489756 13237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:49:36.489869 13237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:36.489943 13237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:36.490024 13237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:49:36.490084 13237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:49:36.500046 13237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46161/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38013
--ranger_config_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:36.501353 13237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:36.502642 13237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:36.513424 13242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:36.513547 13245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:36.513581 13243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:36.514109 13237 server_base.cc:1034] running on GCE node
I20260929 20:49:36.514747 13237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:36.516085 13237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:36.517275 13237 hybrid_clock.cc:648] HybridClock initialized: now 1790714976517223 us; error 46 us; skew 500 ppm
I20260929 20:49:36.517586 13237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:36 dist-test-slave-blv7 krb5kdc[12259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:36.520445 13237 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:49:36.521828 13237 webserver.cc:460] Webserver started at http://127.0.97.62:42571/ using document root <none> and password file <none>
I20260929 20:49:36.522430 13237 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:36.522540 13237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 20:49:36 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:49:36.584044 13237 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:49:36.586912 13253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:36.588299 13237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:49:36.588467 13237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "31805551e99449429984026ee29db309"
format_stamp: "Formatted at 2026-09-29 20:49:32 on dist-test-slave-blv7"
server_key: "bd6cf147286f40a2f920606589033f4d"
server_key_iv: "6351dae8e8defbc353997d344c6418f6"
server_key_version: "kuduclusterkey@0"
I20260929 20:49:36.589433 13237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:36.607925 13237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:36.611459 13237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:36.611749 13237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:36.629421 13237 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38013
I20260929 20:49:36.629454 13304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38013 every 8 connection(s)
I20260929 20:49:36.630971 13237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:49:36.633136 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 13237
I20260929 20:49:36.634110 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41199
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38013
--webserver_port=46487
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46161/kms with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:49:36.639835 13305 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:49:36.643616 13305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309: Bootstrap starting.
I20260929 20:49:36.661674 13305 log.cc:824] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309: Log is configured to *not* fsync() on all Append() calls
I20260929 20:49:36.670210 13305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:49:36.670693 13305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309: Bootstrap complete.
I20260929 20:49:36.679208 13305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31805551e99449429984026ee29db309" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38013 } }
I20260929 20:49:36.680078 13305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31805551e99449429984026ee29db309, State: Initialized, Role: FOLLOWER
I20260929 20:49:36.680650 13305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31805551e99449429984026ee29db309" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38013 } }
I20260929 20:49:36.680845 13305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:49:36.680950 13305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:49:36.681109 13305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:49:36.684789 13305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31805551e99449429984026ee29db309" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38013 } }
I20260929 20:49:36.685242 13305 leader_election.cc:302] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31805551e99449429984026ee29db309; no voters:
I20260929 20:49:36.685814 13305 leader_election.cc:288] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 20:49:36.685956 13309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:49:36.686937 13309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [term 2 LEADER]: Becoming Leader. State: Replica: 31805551e99449429984026ee29db309, State: Running, Role: LEADER
I20260929 20:49:36.687563 13309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31805551e99449429984026ee29db309" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38013 } }
I20260929 20:49:36.687884 13305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:49:36.689896 13310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "31805551e99449429984026ee29db309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31805551e99449429984026ee29db309" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38013 } } }
I20260929 20:49:36.690269 13310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [sys.catalog]: This master's current role is: LEADER
I20260929 20:49:36.692523 13311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31805551e99449429984026ee29db309. Latest consensus state: current_term: 2 leader_uuid: "31805551e99449429984026ee29db309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31805551e99449429984026ee29db309" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38013 } } }
I20260929 20:49:36.692909 13311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309 [sys.catalog]: This master's current role is: LEADER
I20260929 20:49:36.694577 13315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:49:36.700490 13315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:49:36.702139 13315 catalog_manager.cc:1259] Loaded cluster ID: 2bf6635840734132af94c42d5f8f3b9c
I20260929 20:49:36.702296 13315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:49:36.706393 13315 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:49:36.707954 13315 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309: Loaded TSK: 0
I20260929 20:49:36.709589 13315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:49:36.710047 13305 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260929 20:49:37.053602 13307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:37.054154 13307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:37.054345 13307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:37.065514 13307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:37.065650 13307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:37.065901 13307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:49:37.081111 13307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=46487
--tserver_master_addrs=127.0.97.62:38013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:37.082846 13307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:37.084587 13307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:37.138924 13343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:37.140337 13345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:37.141803 13342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:37.143265 13307 server_base.cc:1034] running on GCE node
I20260929 20:49:37.144093 13307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:37.155299 13307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:37.156644 13307 hybrid_clock.cc:648] HybridClock initialized: now 1790714977156551 us; error 89 us; skew 500 ppm
I20260929 20:49:37.157120 13307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:37 dist-test-slave-blv7 krb5kdc[12259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:37.161672 13307 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:49:37.189832 13307 webserver.cc:460] Webserver started at http://127.0.97.1:46487/ using document root <none> and password file <none>
I20260929 20:49:37.190946 13307 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:37.191237 13307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 20:49:37 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:49:37.239861 13307 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.010s sys 0.000s
I20260929 20:49:37.248592 13354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:37.250483 13307 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.006s sys 0.002s
I20260929 20:49:37.250731 13307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "d3151417daca4f54860975b65dbb83c4"
format_stamp: "Formatted at 2026-09-29 20:49:33 on dist-test-slave-blv7"
server_key: "fbf3acbb308aa62106e5be36671f155d"
server_key_iv: "b1efacd110d0a21d9328dcd5a135ee94"
server_key_version: "kuduclusterkey@0"
I20260929 20:49:37.252203 13307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:37.280972 13307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:37.286051 13307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:37.286353 13307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:37.287598 13307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:49:37.289582 13307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:49:37.289691 13307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:37.289805 13307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:49:37.289885 13307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:37.359339 13307 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41199
I20260929 20:49:37.359439 13466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41199 every 8 connection(s)
I20260929 20:49:37.360749 13307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:49:37.366503 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 13307
I20260929 20:49:37.367326 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:43541
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38013
--webserver_port=44419
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46161/kms with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:49:37 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:49:37.406046 13470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:37.364346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38013 (local address 127.0.97.1:50055)
0929 20:49:37.371591 (+ 7245us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:37.371648 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:37.373901 (+ 2253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:37.375616 (+ 1715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:37.375628 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:37.376261 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:49:37.376990 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:37.377026 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:37.380806 (+ 3780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:37.380813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:37.384404 (+ 3591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:37.384425 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:37.386408 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:37.386416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:37.386700 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:37.392348 (+ 5648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:49:37.392378 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:49:37.395524 (+ 3146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:49:37.400756 (+ 5232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:37.400767 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:37.400785 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:37.401374 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:37.401939 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:37.401945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:37.401949 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:37.402208 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:37.403362 (+ 1154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:49:37.403371 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:49:37.403777 (+ 406us) client_negotiation.cc:769] Sending connection context
0929 20:49:37.404072 (+ 295us) client_negotiation.cc:240] Negotiation successful
0929 20:49:37.404605 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6949,"thread_start_us":6731,"threads_started":1}
I20260929 20:49:37.408397 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:37.364090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50055 (local address 127.0.97.62:38013)
0929 20:49:37.364761 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:37.364778 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:37.372230 (+ 7452us) server_negotiation.cc:407] Connection header received
0929 20:49:37.374527 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:37.374541 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:37.374968 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:37.375430 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:37.377237 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:37.379199 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:37.384647 (+ 5448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:37.385783 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:37.385991 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:37.395916 (+ 9925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:37.395927 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:37.395951 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:37.396007 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:37.400571 (+ 4564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:37.401555 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:37.401578 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:37.401588 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:37.401778 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:37.402358 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:37.402363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:37.402367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:37.403099 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:37.403274 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:37.406737 (+ 3463us) server_negotiation.cc:299] Negotiation successful
0929 20:49:37.407184 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":252,"threads_started":1}
I20260929 20:49:37.412854 13467 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38013
I20260929 20:49:37.413179 13467 heartbeater.cc:464] Registering TS with master...
I20260929 20:49:37.414069 13467 heartbeater.cc:511] Master 127.0.97.62:38013 requested a full tablet report, sending...
W20260929 20:49:37.417205 13467 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38013 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 20:49:37.417426 13467 heartbeater.cc:464] Registering TS with master...
I20260929 20:49:37.417688 13467 heartbeater.cc:511] Master 127.0.97.62:38013 requested a full tablet report, sending...
I20260929 20:49:37.418974 13467 heartbeater.cc:464] Registering TS with master...
I20260929 20:49:37.419309 13467 heartbeater.cc:511] Master 127.0.97.62:38013 requested a full tablet report, sending...
W20260929 20:49:37.420352 13467 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 20:49:37.568679 13478 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260929 20:49:37.796511 13471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:37.796880 13471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:37.797106 13471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:37.806227 13471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:37.806389 13471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:37.806598 13471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:49:37.819200 13471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:43541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=44419
--tserver_master_addrs=127.0.97.62:38013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:37.821151 13471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:37.822942 13471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:37.846591 13485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:37.847491 13486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:37.854425 13489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:37.855221 13471 server_base.cc:1034] running on GCE node
I20260929 20:49:37.855970 13471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:37.857161 13471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:37.858455 13471 hybrid_clock.cc:648] HybridClock initialized: now 1790714977858382 us; error 58 us; skew 500 ppm
I20260929 20:49:37.858861 13471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:37 dist-test-slave-blv7 krb5kdc[12259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:37.863089 13471 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:49:37.874835 13471 webserver.cc:460] Webserver started at http://127.0.97.2:44419/ using document root <none> and password file <none>
I20260929 20:49:37.875802 13471 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:37.875973 13471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 20:49:37 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:49:37.918781 13471 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.011s sys 0.000s
I20260929 20:49:37.922678 13498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:37.924641 13471 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:49:37.924893 13471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "522b57e621e845e6aa9f565598e39825"
format_stamp: "Formatted at 2026-09-29 20:49:34 on dist-test-slave-blv7"
server_key: "b5aebcc339adb785fe858866b9548636"
server_key_iv: "1054bd899cf2cbef5be968ea2e8dc2f2"
server_key_version: "kuduclusterkey@0"
I20260929 20:49:37.926369 13471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 29 20:49:37 dist-test-slave-blv7 krb5kdc[12259](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790714977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:37.954542 13471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:37.960052 13471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:37.960409 13471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:37.961699 13471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:49:37.963797 13471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:49:37.963940 13471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 20:49:37.964064 13471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:49:37.964152 13471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:38.012889 13471 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43541
I20260929 20:49:38.012950 13611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43541 every 8 connection(s)
I20260929 20:49:38.014676 13471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:49:38.015857 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 13471
I20260929 20:49:38.016649 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:46161/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35395
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38013
--webserver_port=43961
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46161/kms with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:49:38 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:49:38.059403 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:38.026315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50839 (local address 127.0.97.62:38013)
0929 20:49:38.037496 (+ 11181us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:38.037502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:38.038301 (+ 799us) server_negotiation.cc:407] Connection header received
0929 20:49:38.041053 (+ 2752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:38.041060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:38.041152 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:38.041238 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:38.043631 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.044777 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.047225 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.047890 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.047993 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:38.052312 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:38.052319 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:38.052324 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:38.052364 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:38.055783 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:38.056774 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:38.056780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:38.056784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:38.056936 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:38.057424 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:38.057430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:38.057433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:38.057773 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:38.057898 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:38.058796 (+ 898us) server_negotiation.cc:299] Negotiation successful
0929 20:49:38.059008 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11050,"thread_start_us":10899,"threads_started":1}
I20260929 20:49:38.061846 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:38.026050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38013 (local address 127.0.97.2:50839)
0929 20:49:38.038103 (+ 12053us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:38.038161 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:38.040674 (+ 2513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:38.041873 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:38.041886 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:38.042530 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:49:38.043414 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:38.043441 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.044976 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.044981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:38.047012 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:38.047061 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.048334 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.048341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:38.048646 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:38.049387 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:49:38.049417 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:49:38.052068 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:49:38.055958 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:38.055967 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:38.055984 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:38.056642 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:38.057064 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:38.057068 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:38.057072 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:38.057311 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:38.057984 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:49:38.057992 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:49:38.058381 (+ 389us) client_negotiation.cc:769] Sending connection context
0929 20:49:38.059927 (+ 1546us) client_negotiation.cc:240] Negotiation successful
0929 20:49:38.060487 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11737,"thread_start_us":11522,"threads_started":1}
I20260929 20:49:38.063709 13612 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38013
I20260929 20:49:38.063961 13612 heartbeater.cc:464] Registering TS with master...
I20260929 20:49:38.064740 13612 heartbeater.cc:511] Master 127.0.97.62:38013 requested a full tablet report, sending...
I20260929 20:49:38.066840 13270 ts_manager.cc:194] Registered new tserver with Master: 522b57e621e845e6aa9f565598e39825 (127.0.97.2:43541)
I20260929 20:49:38.069007 13270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50839
Sep 29 20:49:38 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:49:38.420492 13467 heartbeater.cc:464] Registering TS with master...
I20260929 20:49:38.420910 13467 heartbeater.cc:511] Master 127.0.97.62:38013 requested a full tablet report, sending...
I20260929 20:49:38.422199 13270 ts_manager.cc:194] Registered new tserver with Master: d3151417daca4f54860975b65dbb83c4 (127.0.97.1:41199)
I20260929 20:49:38.423316 13270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50055
W20260929 20:49:38.438635 13614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:38.439034 13614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:38.439236 13614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:38.449980 13614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:38.450199 13614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:38.450460 13614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:49:38.465785 13614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:46161/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43961
--tserver_master_addrs=127.0.97.62:38013
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:38.467787 13614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:38.469643 13614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:38.486236 13621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:38.488368 13622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:38.492292 13624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:38.492810 13614 server_base.cc:1034] running on GCE node
I20260929 20:49:38.493629 13614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:38.494884 13614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:38.496199 13614 hybrid_clock.cc:648] HybridClock initialized: now 1790714978496130 us; error 75 us; skew 500 ppm
I20260929 20:49:38.496591 13614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:38 dist-test-slave-blv7 krb5kdc[12259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:38.500352 13614 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:49:38.502017 13614 webserver.cc:460] Webserver started at http://127.0.97.3:43961/ using document root <none> and password file <none>
I20260929 20:49:38.502698 13614 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:38.502821 13614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 20:49:38 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:49:38.524504 13614 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:49:38.527611 13632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:38.529304 13614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:49:38.529531 13614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "4a80a03682ef4d82abc86e912b6a7a58"
format_stamp: "Formatted at 2026-09-29 20:49:35 on dist-test-slave-blv7"
server_key: "536a791d9ff7ab1eca3c0ce928f25716"
server_key_iv: "5c9959049301889d11905bfadad08be6"
server_key_version: "kuduclusterkey@0"
I20260929 20:49:38.531044 13614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:38.567772 13614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:38.573134 13614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:38.573457 13614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:38.574695 13614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:49:38.576725 13614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:49:38.576843 13614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:38.576956 13614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:49:38.577049 13614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:38.624185 13614 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35395
I20260929 20:49:38.624290 13744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35395 every 8 connection(s)
I20260929 20:49:38.626098 13614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:49:38.633610 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 13614
Sep 29 20:49:38 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:49:38.674785 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:38.635304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46363 (local address 127.0.97.62:38013)
0929 20:49:38.651183 (+ 15879us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:38.651189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:38.651207 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:49:38.651276 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:38.651281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:38.651383 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:38.651505 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:38.657929 (+ 6424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.659188 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.661753 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.662402 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.662503 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:38.667185 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:38.667193 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:38.667199 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:38.667243 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:38.670758 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:38.671825 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:38.671831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:38.671835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:38.671986 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:38.672527 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:38.672532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:38.672535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:38.672889 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:38.673062 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:38.674058 (+ 996us) server_negotiation.cc:299] Negotiation successful
0929 20:49:38.674357 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15766,"thread_start_us":15637,"threads_started":1}
I20260929 20:49:38.676944 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:38.631336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38013 (local address 127.0.97.3:46363)
0929 20:49:38.647705 (+ 16369us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:38.647775 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:38.650404 (+ 2629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:38.655437 (+ 5033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:38.655452 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:38.656084 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:49:38.656957 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:38.656985 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.659416 (+ 2431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.659421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:38.661515 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:38.661531 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.662884 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.662892 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:38.663217 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:38.664028 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:49:38.664066 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:49:38.666810 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:49:38.670953 (+ 4143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:38.670963 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:38.670983 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:38.671679 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:38.672152 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:38.672157 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:38.672161 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:38.672400 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:38.673165 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:49:38.673174 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:49:38.673624 (+ 450us) client_negotiation.cc:769] Sending connection context
0929 20:49:38.674911 (+ 1287us) client_negotiation.cc:240] Negotiation successful
0929 20:49:38.675515 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15968,"thread_start_us":15725,"threads_started":1}
I20260929 20:49:38.679325 13745 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38013
I20260929 20:49:38.679734 13745 heartbeater.cc:464] Registering TS with master...
I20260929 20:49:38.680684 13745 heartbeater.cc:511] Master 127.0.97.62:38013 requested a full tablet report, sending...
I20260929 20:49:38.682525 13270 ts_manager.cc:194] Registered new tserver with Master: 4a80a03682ef4d82abc86e912b6a7a58 (127.0.97.3:35395)
I20260929 20:49:38.683297 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:38.651136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52726 (local address 127.0.97.62:38013)
0929 20:49:38.660586 (+ 9450us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:38.660592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:38.660610 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:49:38.660673 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:38.660678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:38.660757 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:38.660875 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:38.663237 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.664765 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.666716 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.667453 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.671599 (+ 4146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:38.671901 (+ 302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:38.671908 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:38.671913 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:38.671956 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:38.675726 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:38.677399 (+ 1673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:38.677405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:38.677409 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:38.677570 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:38.680640 (+ 3070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:38.680646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:38.680648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:38.681017 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:38.681269 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:38.682625 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0929 20:49:38.682868 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9313,"thread_start_us":9894,"threads_started":1}
I20260929 20:49:38.683809 13270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46363
I20260929 20:49:38.688424 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:49:38.712097 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:38.692106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52728 (local address 127.0.97.62:38013)
0929 20:49:38.692293 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:38.692299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:38.694915 (+ 2616us) server_negotiation.cc:407] Connection header received
0929 20:49:38.695184 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:38.695191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:38.695277 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:38.695377 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:38.697628 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.698729 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.700741 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.701314 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.701413 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:38.705564 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:38.705571 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:38.705576 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:38.705619 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:38.709083 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:38.709944 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:38.709949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:38.709953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:38.710086 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:38.710489 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:38.710495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:38.710499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:38.710860 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:38.711123 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:38.711497 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 20:49:38.711741 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:49:38.718556 13270 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:49:38.730345 13270 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 29 20:49:38 dist-test-slave-blv7 krb5kdc[12259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714977, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 20:49:38.770571 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:38.756483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43541 (local address 127.0.0.1:52304)
0929 20:49:38.757135 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:38.757180 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:38.757350 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:38.763925 (+ 6575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:38.763933 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:38.763963 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:49:38.764316 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:38.764327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.765593 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.765599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:38.767769 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:38.767792 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.769634 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.769641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:38.769858 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:38.769920 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 20:49:38.770048 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 20:49:38.770239 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":678,"threads_started":1}
I20260929 20:49:38.771806 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:38.756991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52304 (local address 127.0.97.2:43541)
0929 20:49:38.763414 (+ 6423us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:38.763422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:38.763445 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:49:38.763531 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:38.763536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:38.763724 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:38.763898 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:49:38.764478 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.765461 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.767959 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.769363 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.771063 (+ 1700us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:38.771260 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:38.771330 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 20:49:38.771495 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6272,"thread_start_us":6166,"threads_started":1}
I20260929 20:49:38.780314 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:38.761245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49482 (local address 127.0.97.1:41199)
0929 20:49:38.761880 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:38.761891 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:38.769999 (+ 8108us) server_negotiation.cc:407] Connection header received
0929 20:49:38.770319 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:38.770329 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:38.770605 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:38.770792 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:49:38.771733 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.773488 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.777251 (+ 3763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.778819 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.778923 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:38.779152 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:38.779794 (+ 642us) server_negotiation.cc:299] Negotiation successful
0929 20:49:38.779955 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":574,"threads_started":1}
I20260929 20:49:38.780453 13546 tablet_service.cc:1467] Processing CreateTablet for tablet 43b63973ed4d439b93c9aafa5078b825 (DEFAULT_TABLE table=test-table [id=7f7628c4f02b41e7b0c94cdfe7fca5f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:49:38.781287 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:38.760735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41199 (local address 127.0.0.1:49482)
0929 20:49:38.769882 (+ 9147us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:38.769899 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:38.770110 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:38.771104 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:38.771110 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:38.771137 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:49:38.771488 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:38.771499 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.774900 (+ 3401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.774906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:38.777061 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:38.777077 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.779374 (+ 2297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.779381 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:38.779581 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:38.779617 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 20:49:38.780726 (+ 1109us) client_negotiation.cc:240] Negotiation successful
0929 20:49:38.780873 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9028,"thread_start_us":8894,"threads_started":1}
I20260929 20:49:38.781935 13546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b63973ed4d439b93c9aafa5078b825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:49:38.792019 13402 tablet_service.cc:1467] Processing CreateTablet for tablet 43b63973ed4d439b93c9aafa5078b825 (DEFAULT_TABLE table=test-table [id=7f7628c4f02b41e7b0c94cdfe7fca5f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:49:38.797480 13402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b63973ed4d439b93c9aafa5078b825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:49:38.809939 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:38.775243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35395 (local address 127.0.0.1:41632)
0929 20:49:38.775389 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:38.775407 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:38.775506 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:38.795536 (+ 20030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:38.795542 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:38.795572 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:49:38.795778 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:38.795789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.803988 (+ 8199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.803995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:38.806012 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:38.806040 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.809193 (+ 3153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.809201 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:38.809401 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:38.809443 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 20:49:38.809513 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 20:49:38.809668 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 20:49:38.811357 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:38.770380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41632 (local address 127.0.97.3:35395)
0929 20:49:38.793230 (+ 22850us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:38.793239 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:38.793273 (+ 34us) server_negotiation.cc:407] Connection header received
0929 20:49:38.793387 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:38.793395 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:38.795325 (+ 1930us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:38.795536 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:49:38.795982 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.803821 (+ 7839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.807384 (+ 3563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.808986 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.810505 (+ 1519us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:38.810741 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:38.810816 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 20:49:38.810976 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22668,"thread_start_us":34557,"threads_started":1}
I20260929 20:49:38.815578 13680 tablet_service.cc:1467] Processing CreateTablet for tablet 43b63973ed4d439b93c9aafa5078b825 (DEFAULT_TABLE table=test-table [id=7f7628c4f02b41e7b0c94cdfe7fca5f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:49:38.816624 13766 tablet_bootstrap.cc:492] T 43b63973ed4d439b93c9aafa5078b825 P 522b57e621e845e6aa9f565598e39825: Bootstrap starting.
I20260929 20:49:38.817196 13680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b63973ed4d439b93c9aafa5078b825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:49:38.820238 13766 tablet_bootstrap.cc:654] T 43b63973ed4d439b93c9aafa5078b825 P 522b57e621e845e6aa9f565598e39825: Neither blocks nor log segments found. Creating new log.
I20260929 20:49:38.821524 13766 log.cc:824] T 43b63973ed4d439b93c9aafa5078b825 P 522b57e621e845e6aa9f565598e39825: Log is configured to *not* fsync() on all Append() calls
I20260929 20:49:38.833446 13766 tablet_bootstrap.cc:492] T 43b63973ed4d439b93c9aafa5078b825 P 522b57e621e845e6aa9f565598e39825: No bootstrap required, opened a new log
I20260929 20:49:38.833868 13766 ts_tablet_manager.cc:1397] T 43b63973ed4d439b93c9aafa5078b825 P 522b57e621e845e6aa9f565598e39825: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.007s
I20260929 20:49:38.846019 13766 raft_consensus.cc:348] T 43b63973ed4d439b93c9aafa5078b825 P 522b57e621e845e6aa9f565598e39825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522b57e621e845e6aa9f565598e39825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43541 } } peers { permanent_uuid: "d3151417daca4f54860975b65dbb83c4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41199 } } peers { permanent_uuid: "4a80a03682ef4d82abc86e912b6a7a58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35395 } }
I20260929 20:49:38.846630 13766 raft_consensus.cc:374] T 43b63973ed4d439b93c9aafa5078b825 P 522b57e621e845e6aa9f565598e39825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:49:38.846779 13766 raft_consensus.cc:729] T 43b63973ed4d439b93c9aafa5078b825 P 522b57e621e845e6aa9f565598e39825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522b57e621e845e6aa9f565598e39825, State: Initialized, Role: FOLLOWER
I20260929 20:49:38.847572 13766 consensus_queue.cc:260] T 43b63973ed4d439b93c9aafa5078b825 P 522b57e621e845e6aa9f565598e39825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522b57e621e845e6aa9f565598e39825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43541 } } peers { permanent_uuid: "d3151417daca4f54860975b65dbb83c4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41199 } } peers { permanent_uuid: "4a80a03682ef4d82abc86e912b6a7a58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35395 } }
I20260929 20:49:38.850234 13768 tablet_bootstrap.cc:492] T 43b63973ed4d439b93c9aafa5078b825 P d3151417daca4f54860975b65dbb83c4: Bootstrap starting.
I20260929 20:49:38.856812 13769 tablet_bootstrap.cc:492] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58: Bootstrap starting.
I20260929 20:49:38.859335 13768 tablet_bootstrap.cc:654] T 43b63973ed4d439b93c9aafa5078b825 P d3151417daca4f54860975b65dbb83c4: Neither blocks nor log segments found. Creating new log.
I20260929 20:49:38.860319 13769 tablet_bootstrap.cc:654] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58: Neither blocks nor log segments found. Creating new log.
I20260929 20:49:38.861507 13769 log.cc:824] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58: Log is configured to *not* fsync() on all Append() calls
I20260929 20:49:38.861959 13768 log.cc:824] T 43b63973ed4d439b93c9aafa5078b825 P d3151417daca4f54860975b65dbb83c4: Log is configured to *not* fsync() on all Append() calls
I20260929 20:49:38.866252 13766 ts_tablet_manager.cc:1428] T 43b63973ed4d439b93c9aafa5078b825 P 522b57e621e845e6aa9f565598e39825: Time spent starting tablet: real 0.032s user 0.019s sys 0.009s
I20260929 20:49:38.866765 13612 heartbeater.cc:503] Master 127.0.97.62:38013 was elected leader, sending a full tablet report...
I20260929 20:49:38.873489 13768 tablet_bootstrap.cc:492] T 43b63973ed4d439b93c9aafa5078b825 P d3151417daca4f54860975b65dbb83c4: No bootstrap required, opened a new log
I20260929 20:49:38.873829 13768 ts_tablet_manager.cc:1397] T 43b63973ed4d439b93c9aafa5078b825 P d3151417daca4f54860975b65dbb83c4: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.004s
I20260929 20:49:38.875845 13769 tablet_bootstrap.cc:492] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58: No bootstrap required, opened a new log
I20260929 20:49:38.876147 13769 ts_tablet_manager.cc:1397] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20260929 20:49:38.884488 13768 raft_consensus.cc:348] T 43b63973ed4d439b93c9aafa5078b825 P d3151417daca4f54860975b65dbb83c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522b57e621e845e6aa9f565598e39825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43541 } } peers { permanent_uuid: "d3151417daca4f54860975b65dbb83c4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41199 } } peers { permanent_uuid: "4a80a03682ef4d82abc86e912b6a7a58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35395 } }
I20260929 20:49:38.885025 13768 raft_consensus.cc:374] T 43b63973ed4d439b93c9aafa5078b825 P d3151417daca4f54860975b65dbb83c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:49:38.885172 13768 raft_consensus.cc:729] T 43b63973ed4d439b93c9aafa5078b825 P d3151417daca4f54860975b65dbb83c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3151417daca4f54860975b65dbb83c4, State: Initialized, Role: FOLLOWER
I20260929 20:49:38.885988 13768 consensus_queue.cc:260] T 43b63973ed4d439b93c9aafa5078b825 P d3151417daca4f54860975b65dbb83c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522b57e621e845e6aa9f565598e39825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43541 } } peers { permanent_uuid: "d3151417daca4f54860975b65dbb83c4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41199 } } peers { permanent_uuid: "4a80a03682ef4d82abc86e912b6a7a58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35395 } }
I20260929 20:49:38.886466 13769 raft_consensus.cc:348] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522b57e621e845e6aa9f565598e39825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43541 } } peers { permanent_uuid: "d3151417daca4f54860975b65dbb83c4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41199 } } peers { permanent_uuid: "4a80a03682ef4d82abc86e912b6a7a58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35395 } }
I20260929 20:49:38.886916 13769 raft_consensus.cc:374] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:49:38.887063 13769 raft_consensus.cc:729] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a80a03682ef4d82abc86e912b6a7a58, State: Initialized, Role: FOLLOWER
I20260929 20:49:38.887780 13769 consensus_queue.cc:260] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522b57e621e845e6aa9f565598e39825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43541 } } peers { permanent_uuid: "d3151417daca4f54860975b65dbb83c4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41199 } } peers { permanent_uuid: "4a80a03682ef4d82abc86e912b6a7a58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35395 } }
I20260929 20:49:38.899277 13745 heartbeater.cc:503] Master 127.0.97.62:38013 was elected leader, sending a full tablet report...
I20260929 20:49:38.901188 13769 ts_tablet_manager.cc:1428] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58: Time spent starting tablet: real 0.025s user 0.012s sys 0.010s
I20260929 20:49:38.911857 13467 heartbeater.cc:503] Master 127.0.97.62:38013 was elected leader, sending a full tablet report...
I20260929 20:49:38.912700 13768 ts_tablet_manager.cc:1428] T 43b63973ed4d439b93c9aafa5078b825 P d3151417daca4f54860975b65dbb83c4: Time spent starting tablet: real 0.039s user 0.019s sys 0.010s
I20260929 20:49:38.943773 13774 raft_consensus.cc:482] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:49:38.944098 13774 raft_consensus.cc:504] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522b57e621e845e6aa9f565598e39825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43541 } } peers { permanent_uuid: "d3151417daca4f54860975b65dbb83c4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41199 } } peers { permanent_uuid: "4a80a03682ef4d82abc86e912b6a7a58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35395 } }
I20260929 20:49:38.946254 13774 leader_election.cc:288] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 522b57e621e845e6aa9f565598e39825 (127.0.97.2:43541), d3151417daca4f54860975b65dbb83c4 (127.0.97.1:41199)
I20260929 20:49:38.954948 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:38.946504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43541 (local address 127.0.97.3:39023)
0929 20:49:38.947152 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:38.947173 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:38.947310 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:38.948462 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:38.948467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:38.948497 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:49:38.948713 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:38.948723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.950175 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.950181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:38.952323 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:38.952339 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.954136 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.954143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:38.954365 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:38.954410 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 20:49:38.954500 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 20:49:38.954643 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542}
I20260929 20:49:38.955727 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:38.946771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39023 (local address 127.0.97.2:43541)
0929 20:49:38.946974 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:38.946980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:38.947991 (+ 1011us) server_negotiation.cc:407] Connection header received
0929 20:49:38.948081 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:38.948087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:38.948179 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:38.948266 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:49:38.948874 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.950006 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.952531 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.953815 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.953947 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:38.954050 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:38.955307 (+ 1257us) server_negotiation.cc:299] Negotiation successful
0929 20:49:38.955466 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:49:38.962371 13566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b63973ed4d439b93c9aafa5078b825" candidate_uuid: "4a80a03682ef4d82abc86e912b6a7a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522b57e621e845e6aa9f565598e39825" is_pre_election: true
I20260929 20:49:38.962911 13566 raft_consensus.cc:2454] T 43b63973ed4d439b93c9aafa5078b825 P 522b57e621e845e6aa9f565598e39825 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a80a03682ef4d82abc86e912b6a7a58 in term 0.
I20260929 20:49:38.966495 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:38.947758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60217 (local address 127.0.97.1:41199)
0929 20:49:38.947943 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:38.947949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:38.956335 (+ 8386us) server_negotiation.cc:407] Connection header received
0929 20:49:38.956692 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:38.956698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:38.956776 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:38.956835 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:49:38.957451 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.958592 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.964135 (+ 5543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.965359 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.965478 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:38.965582 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:38.966110 (+ 528us) server_negotiation.cc:299] Negotiation successful
0929 20:49:38.966245 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:49:38.967327 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:38.946941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41199 (local address 127.0.97.3:60217)
0929 20:49:38.956242 (+ 9301us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:38.956262 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:38.956518 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:38.957069 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:38.957074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:38.957099 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:49:38.957283 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:38.957293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.958767 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.958772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:38.963944 (+ 5172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:38.963959 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:38.965715 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:38.965721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:38.965924 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:38.965957 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:49:38.966882 (+ 925us) client_negotiation.cc:240] Negotiation successful
0929 20:49:38.967037 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9202,"thread_start_us":14387,"threads_started":1}
I20260929 20:49:38.969619 13635 leader_election.cc:302] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a80a03682ef4d82abc86e912b6a7a58, 522b57e621e845e6aa9f565598e39825; no voters:
I20260929 20:49:38.970287 13774 raft_consensus.cc:2781] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:49:38.970438 13774 raft_consensus.cc:482] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:49:38.970530 13774 raft_consensus.cc:3037] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:49:38.972009 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b63973ed4d439b93c9aafa5078b825" candidate_uuid: "4a80a03682ef4d82abc86e912b6a7a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3151417daca4f54860975b65dbb83c4" is_pre_election: true
I20260929 20:49:38.972551 13422 raft_consensus.cc:2454] T 43b63973ed4d439b93c9aafa5078b825 P d3151417daca4f54860975b65dbb83c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a80a03682ef4d82abc86e912b6a7a58 in term 0.
I20260929 20:49:38.974099 13774 raft_consensus.cc:504] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522b57e621e845e6aa9f565598e39825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43541 } } peers { permanent_uuid: "d3151417daca4f54860975b65dbb83c4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41199 } } peers { permanent_uuid: "4a80a03682ef4d82abc86e912b6a7a58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35395 } }
I20260929 20:49:38.974687 13774 leader_election.cc:288] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58 [CANDIDATE]: Term 1 election: Requested vote from peers 522b57e621e845e6aa9f565598e39825 (127.0.97.2:43541), d3151417daca4f54860975b65dbb83c4 (127.0.97.1:41199)
I20260929 20:49:38.975153 13566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b63973ed4d439b93c9aafa5078b825" candidate_uuid: "4a80a03682ef4d82abc86e912b6a7a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522b57e621e845e6aa9f565598e39825"
I20260929 20:49:38.975379 13566 raft_consensus.cc:3037] T 43b63973ed4d439b93c9aafa5078b825 P 522b57e621e845e6aa9f565598e39825 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:49:38.975898 13422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b63973ed4d439b93c9aafa5078b825" candidate_uuid: "4a80a03682ef4d82abc86e912b6a7a58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3151417daca4f54860975b65dbb83c4"
I20260929 20:49:38.976089 13422 raft_consensus.cc:3037] T 43b63973ed4d439b93c9aafa5078b825 P d3151417daca4f54860975b65dbb83c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:49:38.978562 13566 raft_consensus.cc:2454] T 43b63973ed4d439b93c9aafa5078b825 P 522b57e621e845e6aa9f565598e39825 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a80a03682ef4d82abc86e912b6a7a58 in term 1.
I20260929 20:49:38.979058 13635 leader_election.cc:302] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a80a03682ef4d82abc86e912b6a7a58, 522b57e621e845e6aa9f565598e39825; no voters:
I20260929 20:49:38.979005 13422 raft_consensus.cc:2454] T 43b63973ed4d439b93c9aafa5078b825 P d3151417daca4f54860975b65dbb83c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a80a03682ef4d82abc86e912b6a7a58 in term 1.
I20260929 20:49:38.979393 13774 raft_consensus.cc:2781] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:49:38.981261 13774 raft_consensus.cc:686] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58 [term 1 LEADER]: Becoming Leader. State: Replica: 4a80a03682ef4d82abc86e912b6a7a58, State: Running, Role: LEADER
I20260929 20:49:38.981897 13774 consensus_queue.cc:237] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522b57e621e845e6aa9f565598e39825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43541 } } peers { permanent_uuid: "d3151417daca4f54860975b65dbb83c4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41199 } } peers { permanent_uuid: "4a80a03682ef4d82abc86e912b6a7a58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35395 } }
I20260929 20:49:38.987195 13270 catalog_manager.cc:5495] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a80a03682ef4d82abc86e912b6a7a58 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "4a80a03682ef4d82abc86e912b6a7a58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "522b57e621e845e6aa9f565598e39825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3151417daca4f54860975b65dbb83c4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a80a03682ef4d82abc86e912b6a7a58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35395 } health_report { overall_health: HEALTHY } } }
W20260929 20:49:39.022925 13613 tablet.cc:2369] T 43b63973ed4d439b93c9aafa5078b825 P 522b57e621e845e6aa9f565598e39825: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:49:39.038770 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:39.032035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41634 (local address 127.0.97.3:35395)
0929 20:49:39.032211 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:39.032217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:39.032235 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:49:39.032340 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:39.032359 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:39.032456 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:39.032603 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:49:39.033226 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:39.034761 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:39.036283 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:39.036908 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:39.037051 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:39.037588 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:49:39.038232 (+ 644us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:49:39.038389 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:39.038436 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 20:49:39.038517 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:49:39.047712 13566 raft_consensus.cc:1261] T 43b63973ed4d439b93c9aafa5078b825 P 522b57e621e845e6aa9f565598e39825 [term 1 FOLLOWER]: Refusing update from remote peer 4a80a03682ef4d82abc86e912b6a7a58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:49:39.047812 13422 raft_consensus.cc:1261] T 43b63973ed4d439b93c9aafa5078b825 P d3151417daca4f54860975b65dbb83c4 [term 1 FOLLOWER]: Refusing update from remote peer 4a80a03682ef4d82abc86e912b6a7a58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:49:39.048641 13776 consensus_queue.cc:1035] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522b57e621e845e6aa9f565598e39825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:49:39.049008 13774 consensus_queue.cc:1035] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3151417daca4f54860975b65dbb83c4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:49:39.060570 13786 mvcc.cc:205] Tried to move back new op lower bound from 7334768554169524224 to 7334768553914523648. Current Snapshot: MvccSnapshot[applied={T|T < 7334768554169524224}]
I20260929 20:49:39.083782 13270 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:49:39.084057 13270 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:49:39.087965 13270 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31805551e99449429984026ee29db309: Sending DeleteTablet for 3 replicas of tablet 43b63973ed4d439b93c9aafa5078b825
I20260929 20:49:39.089100 13546 tablet_service.cc:1514] Processing DeleteTablet for tablet 43b63973ed4d439b93c9aafa5078b825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:49:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52304
I20260929 20:49:39.089361 13402 tablet_service.cc:1514] Processing DeleteTablet for tablet 43b63973ed4d439b93c9aafa5078b825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:49:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49482
I20260929 20:49:39.089880 13680 tablet_service.cc:1514] Processing DeleteTablet for tablet 43b63973ed4d439b93c9aafa5078b825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:49:39 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41632
I20260929 20:49:39.090065 13796 tablet_replica.cc:330] T 43b63973ed4d439b93c9aafa5078b825 P 522b57e621e845e6aa9f565598e39825: stopping tablet replica
I20260929 20:49:39.090464 13796 raft_consensus.cc:2229] T 43b63973ed4d439b93c9aafa5078b825 P 522b57e621e845e6aa9f565598e39825 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:49:39.090986 13796 raft_consensus.cc:2258] T 43b63973ed4d439b93c9aafa5078b825 P 522b57e621e845e6aa9f565598e39825 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:49:39.093058 13796 ts_tablet_manager.cc:1905] T 43b63973ed4d439b93c9aafa5078b825 P 522b57e621e845e6aa9f565598e39825: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:49:39.095777 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 13307
I20260929 20:49:39.100800 13796 ts_tablet_manager.cc:1918] T 43b63973ed4d439b93c9aafa5078b825 P 522b57e621e845e6aa9f565598e39825: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:49:39.101068 13796 log.cc:1196] T 43b63973ed4d439b93c9aafa5078b825 P 522b57e621e845e6aa9f565598e39825: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/43b63973ed4d439b93c9aafa5078b825
I20260929 20:49:39.101653 13796 ts_tablet_manager.cc:1939] T 43b63973ed4d439b93c9aafa5078b825 P 522b57e621e845e6aa9f565598e39825: Deleting consensus metadata
I20260929 20:49:39.103565 13256 catalog_manager.cc:4841] TS 522b57e621e845e6aa9f565598e39825 (127.0.97.2:43541): tablet 43b63973ed4d439b93c9aafa5078b825 (table test-table [id=7f7628c4f02b41e7b0c94cdfe7fca5f7]) successfully deleted
I20260929 20:49:39.109669 13797 tablet_replica.cc:330] T 43b63973ed4d439b93c9aafa5078b825 P d3151417daca4f54860975b65dbb83c4: stopping tablet replica
I20260929 20:49:39.110337 13797 raft_consensus.cc:2229] T 43b63973ed4d439b93c9aafa5078b825 P d3151417daca4f54860975b65dbb83c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:49:39.110957 13797 raft_consensus.cc:2258] T 43b63973ed4d439b93c9aafa5078b825 P d3151417daca4f54860975b65dbb83c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:49:39.114722 13798 tablet_replica.cc:330] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58: stopping tablet replica
I20260929 20:49:39.115305 13798 raft_consensus.cc:2229] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:49:39.116076 13798 raft_consensus.cc:2258] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:49:39.118242 13798 ts_tablet_manager.cc:1905] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:49:39.119397 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:39.097083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.0.97.1:41199)
0929 20:49:39.097258 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:39.097264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:39.097284 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:49:39.097446 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:39.097452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:39.097540 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:39.097645 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:39.103898 (+ 6253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:39.105055 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:39.107003 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:39.107747 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:39.107873 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:39.111229 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:39.111245 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:39.111264 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:39.111320 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:39.115139 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:39.115993 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:39.116003 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:39.116008 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:39.116153 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:39.116791 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:39.116796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:39.116800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:39.117291 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:39.117596 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:39.118735 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0929 20:49:39.118951 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:49:39.120079 13462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:49:39.126792 13798 ts_tablet_manager.cc:1918] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:49:39.127498 13798 log.cc:1196] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/43b63973ed4d439b93c9aafa5078b825
I20260929 20:49:39.128115 13798 ts_tablet_manager.cc:1939] T 43b63973ed4d439b93c9aafa5078b825 P 4a80a03682ef4d82abc86e912b6a7a58: Deleting consensus metadata
I20260929 20:49:39.129881 13256 catalog_manager.cc:4841] TS 4a80a03682ef4d82abc86e912b6a7a58 (127.0.97.3:35395): tablet 43b63973ed4d439b93c9aafa5078b825 (table test-table [id=7f7628c4f02b41e7b0c94cdfe7fca5f7]) successfully deleted
I20260929 20:49:39.120189 13797 ts_tablet_manager.cc:1905] T 43b63973ed4d439b93c9aafa5078b825 P d3151417daca4f54860975b65dbb83c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:49:39.286693 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 13307
W20260929 20:49:39.307075 13256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:41199 (error 108)
I20260929 20:49:39.307247 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 13471
I20260929 20:49:39.309619 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:39.309087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41199 (local address 127.0.0.1:49496)
0929 20:49:39.309316 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:39.309436 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260929 20:49:39.309908 13256 catalog_manager.cc:4584] TS d3151417daca4f54860975b65dbb83c4 (127.0.97.1:41199): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 43b63973ed4d439b93c9aafa5078b825: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41199: connect: Connection refused (error 111)
I20260929 20:49:39.330078 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:39.308335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52308 (local address 127.0.97.2:43541)
0929 20:49:39.308505 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:39.308510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:39.308531 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:49:39.308688 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:39.308695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:39.308790 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:39.308876 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:39.310688 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:39.311775 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:39.313390 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:39.314001 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:39.314138 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:39.322531 (+ 8393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:39.322550 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:39.322569 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:39.322616 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:39.326830 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:39.327740 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:39.327749 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:39.327752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:39.327922 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:39.328344 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:39.328350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:39.328353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:39.328871 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:39.329169 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:39.329506 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 20:49:39.329726 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:49:39.330793 13607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:49:39.363554 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:39.362972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41199 (local address 127.0.0.1:49510)
0929 20:49:39.363268 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:39.363397 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260929 20:49:39.364826 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:39.364421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41199 (local address 127.0.0.1:49512)
0929 20:49:39.364604 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:39.364676 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 20:49:39.425393 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:39.424945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41199 (local address 127.0.0.1:49522)
0929 20:49:39.425148 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:39.425232 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 20:49:39.426425 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:39.426080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41199 (local address 127.0.0.1:49532)
0929 20:49:39.426216 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:39.426300 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 20:49:39.458448 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 13471
I20260929 20:49:39.471961 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 13614
I20260929 20:49:39.485527 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:39.472931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41648 (local address 127.0.97.3:35395)
0929 20:49:39.473149 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:39.473153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:39.473168 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:49:39.473221 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:39.473224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:39.473278 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:39.473385 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:39.474545 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:39.475446 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:39.476687 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:39.477134 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:39.477241 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:39.479809 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:39.479823 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:39.479839 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:39.479870 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:39.482896 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:39.483558 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:39.483565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:39.483567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:39.483724 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:39.484072 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:39.484075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:39.484077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:39.484408 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:39.484693 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:39.484979 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 20:49:39.485192 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:49:39.486264 13740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:49:39.506779 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:39.506343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41199 (local address 127.0.0.1:49536)
0929 20:49:39.506548 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:39.506629 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 20:49:39.507769 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:39.507389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41199 (local address 127.0.0.1:49546)
0929 20:49:39.507571 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:39.507647 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 20:49:39.609018 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 13614
I20260929 20:49:39.624889 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 13237
I20260929 20:49:39.625977 13300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:49:39.734962 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:39.734423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41199 (local address 127.0.0.1:49562)
0929 20:49:39.734665 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:39.734787 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 20:49:39.735185 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 13237
I20260929 20:49:39.755595 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 20:49:40.089713 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 20:49:40.089886 388 mini_ranger.cc:66] Stopping Ranger...
I20260929 20:49:40.424008 388 mini_ranger.cc:68] Stopped Ranger
2026-09-29 20:49:40.424 UTC [12289] LOG: received smart shutdown request
2026-09-29 20:49:40.427 UTC [12289] LOG: background worker "logical replication launcher" (PID 12298) exited with exit code 1
2026-09-29 20:49:40.427 UTC [12293] LOG: shutting down
2026-09-29 20:49:40.584 UTC [12289] LOG: database system is shut down
2026-09-29T20:49:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51490 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:49:40 dist-test-slave-blv7 krb5kdc[13812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:49:40 dist-test-slave-blv7 krb5kdc[13812](info): set up 2 sockets
Sep 29 20:49:40 dist-test-slave-blv7 krb5kdc[13812](info): commencing operation
krb5kdc: starting...
W20260929 20:49:41.691098 388 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:49:41 dist-test-slave-blv7 krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:49:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:49:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:49:41.894613 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43069
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43069,127.0.97.61:40947,127.0.97.60:40293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:49:42.210577 13825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:42.210896 13825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:42.211056 13825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:42.218282 13825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:49:42.218398 13825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:42.218479 13825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:42.218554 13825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:49:42.218614 13825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:49:42.228580 13825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43069,127.0.97.61:40947,127.0.97.60:40293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:42.229894 13825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:42.231187 13825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:42.242625 13833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:42.242672 13831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:42.242694 13830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:42.243347 13825 server_base.cc:1034] running on GCE node
I20260929 20:49:42.244066 13825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:42.245407 13825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:42.246614 13825 hybrid_clock.cc:648] HybridClock initialized: now 1790714982246577 us; error 34 us; skew 500 ppm
I20260929 20:49:42.246929 13825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:42 dist-test-slave-blv7 krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:42.249830 13825 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:49:42.251175 13825 webserver.cc:460] Webserver started at http://127.0.97.62:44513/ using document root <none> and password file <none>
I20260929 20:49:42.251802 13825 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:42.251905 13825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:49:42.252189 13825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:49:42.255153 13825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a"
format_stamp: "Formatted at 2026-09-29 20:49:42 on dist-test-slave-blv7"
server_key: "b8843f2660e1273c375553d6dedf9551"
server_key_iv: "348fcfbe97151fbcba8d0a8cd6baa397"
server_key_version: "encryptionkey@0"
I20260929 20:49:42.255935 13825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a"
format_stamp: "Formatted at 2026-09-29 20:49:42 on dist-test-slave-blv7"
server_key: "b8843f2660e1273c375553d6dedf9551"
server_key_iv: "348fcfbe97151fbcba8d0a8cd6baa397"
server_key_version: "encryptionkey@0"
I20260929 20:49:42.260804 13825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:49:42.264042 13840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:42.265282 13825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:49:42.265427 13825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a"
format_stamp: "Formatted at 2026-09-29 20:49:42 on dist-test-slave-blv7"
server_key: "b8843f2660e1273c375553d6dedf9551"
server_key_iv: "348fcfbe97151fbcba8d0a8cd6baa397"
server_key_version: "encryptionkey@0"
I20260929 20:49:42.265609 13825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:42.295311 13825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:42.299330 13825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:42.299603 13825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:42.316541 13825 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43069
I20260929 20:49:42.316583 13891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43069 every 8 connection(s)
I20260929 20:49:42.317867 13825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:49:42.321472 13892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:49:42.323768 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 13825
I20260929 20:49:42.323942 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:49:42.324530 388 external_mini_cluster.cc:1428] Setting key 92ae150c4acb0d161d7f79fcf4f5bf7b
I20260929 20:49:42.328092 13892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 29 20:49:42 dist-test-slave-blv7 krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:49:42.354579 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:42.331279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56480 (local address 127.0.97.62:43069)
0929 20:49:42.333404 (+ 2125us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:42.333411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:42.333469 (+ 58us) server_negotiation.cc:407] Connection header received
0929 20:49:42.335614 (+ 2145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:42.335622 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:42.335950 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:42.336201 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:42.337748 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:42.339007 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:42.341064 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:42.341634 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:42.341763 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:42.347486 (+ 5723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:42.347496 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:42.347512 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:42.347548 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:42.350758 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:42.351429 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:42.351438 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:42.351441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:42.351630 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:42.351967 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:42.351978 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:42.351982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:42.352592 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:42.352781 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:42.353294 (+ 513us) server_negotiation.cc:299] Negotiation successful
0929 20:49:42.353637 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":1712,"spinlock_wait_cycles":42880,"thread_start_us":1488,"threads_started":1}
I20260929 20:49:42.354646 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:42.331279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43069 (local address 127.0.0.1:56480)
0929 20:49:42.332179 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:42.332228 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:42.335105 (+ 2877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:42.336280 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:42.336288 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:42.337089 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:49:42.337522 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:42.337556 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:42.339183 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:42.339188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:42.340917 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:42.340932 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:42.341877 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:42.341885 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:42.342065 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:42.342699 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:49:42.342728 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:49:42.347245 (+ 4517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:49:42.350873 (+ 3628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:42.350882 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:42.350892 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:42.351313 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:42.351730 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:42.351734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:42.351736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:42.351877 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:42.352702 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:49:42.352708 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:49:42.352909 (+ 201us) client_negotiation.cc:769] Sending connection context
0929 20:49:42.353107 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 20:49:42.353484 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":389,"threads_started":1}
I20260929 20:49:42.357473 13892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:49:42.358400 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:42.358007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40947 (local address 127.0.0.1:38864)
0929 20:49:42.358169 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:42.358286 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 20:49:42.358626 13844 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40947: connect: Connection refused (error 111)
I20260929 20:49:42.360071 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:42.359714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40947 (local address 127.0.0.1:38870)
0929 20:49:42.359844 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:42.359925 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 20:49:42.360240 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40947: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40947: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 20:49:42.372084 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40947
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43069,127.0.97.61:40947,127.0.97.60:40293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:49:42.409586 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } attempt: 1
I20260929 20:49:42.410776 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:42.410339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40947 (local address 127.0.0.1:38886)
0929 20:49:42.410528 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:42.410635 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 20:49:42.411899 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:42.411586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40947 (local address 127.0.0.1:38898)
0929 20:49:42.411703 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:42.411771 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 20:49:42.412072 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40947: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40947: connect: Connection refused (error 111)
I20260929 20:49:42.480304 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } attempt: 2
I20260929 20:49:42.481406 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:42.480944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40947 (local address 127.0.0.1:38902)
0929 20:49:42.481181 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:42.481271 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 20:49:42.482334 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:42.481998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40947 (local address 127.0.0.1:38906)
0929 20:49:42.482131 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:42.482209 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 20:49:42.482499 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40947: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40947: connect: Connection refused (error 111)
I20260929 20:49:42.573717 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } attempt: 3
I20260929 20:49:42.574787 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:42.574371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40947 (local address 127.0.0.1:38914)
0929 20:49:42.574561 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:42.574649 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 20:49:42.575762 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:42.575398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40947 (local address 127.0.0.1:38916)
0929 20:49:42.575561 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:42.575629 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 20:49:42.575927 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40947: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40947: connect: Connection refused (error 111)
W20260929 20:49:42.683888 13899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:42.684196 13899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:42.684329 13899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:42.691664 13899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:49:42.691781 13899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:42.691857 13899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:42.691931 13899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:49:42.691987 13899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:49:42.701817 13899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43069,127.0.97.61:40947,127.0.97.60:40293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:42.703306 13899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:42.704706 13899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:42.715723 13904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:42.715807 13907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:42.715754 13905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:42.716605 13899 server_base.cc:1034] running on GCE node
I20260929 20:49:42.717269 13899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:42.718518 13899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:42.719110 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } attempt: 4
I20260929 20:49:42.719718 13899 hybrid_clock.cc:648] HybridClock initialized: now 1790714982719669 us; error 44 us; skew 500 ppm
I20260929 20:49:42.720110 13899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:49:42.720256 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:42.719677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40947 (local address 127.0.0.1:38924)
0929 20:49:42.719987 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:42.720089 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260929 20:49:42.721349 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:42.720919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40947 (local address 127.0.0.1:38940)
0929 20:49:42.721099 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:42.721203 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 20:49:42.721555 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40947: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40947: connect: Connection refused (error 111)
Sep 29 20:49:42 dist-test-slave-blv7 krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:42.726378 13899 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 20:49:42.727877 13899 webserver.cc:460] Webserver started at http://127.0.97.61:37767/ using document root <none> and password file <none>
I20260929 20:49:42.728550 13899 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:42.728662 13899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:49:42.728933 13899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:49:42.731879 13899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-1/data/instance:
uuid: "b559d96ad22d4c83bd3ae6051d8ecff0"
format_stamp: "Formatted at 2026-09-29 20:49:42 on dist-test-slave-blv7"
server_key: "b13d066f3d18883d864a102be251568b"
server_key_iv: "8fee3ef508107afda9b5ca76981ec864"
server_key_version: "encryptionkey@0"
I20260929 20:49:42.732625 13899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-1/wal/instance:
uuid: "b559d96ad22d4c83bd3ae6051d8ecff0"
format_stamp: "Formatted at 2026-09-29 20:49:42 on dist-test-slave-blv7"
server_key: "b13d066f3d18883d864a102be251568b"
server_key_iv: "8fee3ef508107afda9b5ca76981ec864"
server_key_version: "encryptionkey@0"
I20260929 20:49:42.737460 13899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:49:42.741184 13914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:42.742754 13899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:49:42.742923 13899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-1/wal
uuid: "b559d96ad22d4c83bd3ae6051d8ecff0"
format_stamp: "Formatted at 2026-09-29 20:49:42 on dist-test-slave-blv7"
server_key: "b13d066f3d18883d864a102be251568b"
server_key_iv: "8fee3ef508107afda9b5ca76981ec864"
server_key_version: "encryptionkey@0"
I20260929 20:49:42.743160 13899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:42.774259 13899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:42.778029 13899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:42.778249 13899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:42.794636 13899 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40947
I20260929 20:49:42.794672 13965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40947 every 8 connection(s)
I20260929 20:49:42.795943 13899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-1/data/info.pb
I20260929 20:49:42.799448 13966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:49:42.800658 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 13899
I20260929 20:49:42.800814 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-1/wal/instance
I20260929 20:49:42.801206 388 external_mini_cluster.cc:1428] Setting key 9b172c451732a217ac603a01c87b7ca1
I20260929 20:49:42.805831 13966 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 29 20:49:42 dist-test-slave-blv7 krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:49:42.828512 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:42.808477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.0.97.62:43069)
0929 20:49:42.808696 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:42.808702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:42.809809 (+ 1107us) server_negotiation.cc:407] Connection header received
0929 20:49:42.811707 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:42.811714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:42.811794 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:42.811905 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:42.813374 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:42.814381 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:42.816004 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:42.816647 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:42.816789 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:42.822130 (+ 5341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:42.822137 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:42.822142 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:42.822184 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:42.825312 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:42.826131 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:42.826137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:42.826140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:42.826276 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:42.826665 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:42.826669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:42.826672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:42.827006 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:42.827201 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:42.827959 (+ 758us) server_negotiation.cc:299] Negotiation successful
0929 20:49:42.828191 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 20:49:42.829295 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:42.808861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43069 (local address 127.0.0.1:56486)
0929 20:49:42.809639 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:42.809689 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:42.811430 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:42.812155 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:42.812162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:42.812656 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:49:42.813215 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:42.813231 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:42.814527 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:42.814531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:42.815862 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:42.815874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:42.816849 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:42.816854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:42.817034 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:42.817714 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:49:42.817735 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:49:42.821818 (+ 4083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:49:42.825453 (+ 3635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:42.825462 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:42.825480 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:42.825995 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:42.826369 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:42.826373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:42.826377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:42.826565 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:42.827149 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:49:42.827156 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:49:42.827564 (+ 408us) client_negotiation.cc:769] Sending connection context
0929 20:49:42.827808 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 20:49:42.828203 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":337,"threads_started":1}
I20260929 20:49:42.830945 13966 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:49:42 dist-test-slave-blv7 krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 20:49:42.844875 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:42.831590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40947 (local address 127.0.0.1:38952)
0929 20:49:42.831748 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:42.831766 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:42.831861 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:42.832856 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:42.832859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:42.833202 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:49:42.833336 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:42.833344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:42.835097 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:42.835101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:42.835899 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:42.835909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:42.836993 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:42.836998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:42.837126 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:42.837639 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:49:42.837655 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:49:42.839704 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:49:42.842679 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:42.842683 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:42.842686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:42.843050 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:42.843367 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:42.843370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:42.843373 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:42.843525 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:42.844113 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:49:42.844118 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:49:42.844231 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 20:49:42.844357 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 20:49:42.844595 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 20:49:42.845016 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:42.831813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38952 (local address 127.0.97.61:40947)
0929 20:49:42.832382 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:42.832390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:42.832419 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:49:42.832472 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:42.832480 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:42.832712 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:42.832852 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:42.833476 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:42.834931 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:42.836023 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:42.836852 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:42.836978 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:42.839899 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:42.839910 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:42.839922 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:42.839950 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:42.842550 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:42.843170 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:42.843176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:42.843178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:42.843285 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:42.843629 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:42.843632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:42.843634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:42.843971 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:42.844134 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:42.844468 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 20:49:42.844697 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":393,"thread_start_us":255,"threads_started":1}
I20260929 20:49:42.847333 13966 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:49:42.848526 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:42.848051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40293 (local address 127.0.0.1:58458)
0929 20:49:42.848236 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:42.848366 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.60.keytab.
W20260929 20:49:42.848778 13915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
I20260929 20:49:42.849635 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40293
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43069,127.0.97.61:40947,127.0.97.60:40293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:49:42.850322 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:42.850023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40293 (local address 127.0.0.1:58464)
0929 20:49:42.850156 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:42.850229 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 20:49:42.850514 13966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40293: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
I20260929 20:49:42.899828 13966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } attempt: 1
I20260929 20:49:42.901086 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:42.900548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40293 (local address 127.0.0.1:58478)
0929 20:49:42.900771 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:42.900895 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 20:49:42.902096 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:42.901725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40293 (local address 127.0.0.1:58486)
0929 20:49:42.901880 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:42.901966 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 20:49:42.902297 13966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40293: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
I20260929 20:49:42.970501 13966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } attempt: 2
I20260929 20:49:42.971683 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:42.971166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40293 (local address 127.0.0.1:58502)
0929 20:49:42.971380 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:42.971499 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 20:49:42.972635 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:42.972274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40293 (local address 127.0.0.1:58518)
0929 20:49:42.972409 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:42.972498 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 20:49:42.972846 13966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40293: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
I20260929 20:49:43.020838 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } attempt: 5
Sep 29 20:49:43 dist-test-slave-blv7 krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 20:49:43.033629 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.021680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40947 (local address 127.0.0.1:38956)
0929 20:49:43.021898 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:43.021972 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:43.022118 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:43.022456 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:43.022461 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:43.022846 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:49:43.023067 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:43.023081 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.024103 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.024107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:43.024972 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:43.024984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.025706 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.025712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:43.025893 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:43.026465 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:49:43.026482 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:49:43.028445 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:49:43.031396 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:43.031402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:43.031406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:43.031765 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:43.032103 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:43.032108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:43.032112 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:43.032240 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:43.032726 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:49:43.032731 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:49:43.032872 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 20:49:43.033000 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 20:49:43.033269 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 20:49:43.033629 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.021857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38956 (local address 127.0.97.61:40947)
0929 20:49:43.022048 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:43.022054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:43.022075 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:49:43.022253 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:43.022257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:43.022321 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:43.022433 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:43.023222 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.023964 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.025113 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.025550 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.025674 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:43.028620 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:43.028625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:43.028628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:43.028659 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:43.031282 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.031873 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.031879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.031881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.032002 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.032336 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.032341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.032344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.032615 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:43.032742 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:43.033113 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 20:49:43.033310 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 20:49:43.034915 13892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:49:43.035953 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.035580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40293 (local address 127.0.0.1:58530)
0929 20:49:43.035733 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:43.035810 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 20:49:43.036873 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.036556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40293 (local address 127.0.0.1:58538)
0929 20:49:43.036697 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:43.036776 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 20:49:43.037089 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40293: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
I20260929 20:49:43.064062 13966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } attempt: 3
I20260929 20:49:43.065212 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.064700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40293 (local address 127.0.0.1:58548)
0929 20:49:43.064932 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:43.065035 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 20:49:43.066262 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.065802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40293 (local address 127.0.0.1:58554)
0929 20:49:43.066025 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:43.066112 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260929 20:49:43.066463 13966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40293: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
I20260929 20:49:43.088297 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } attempt: 1
I20260929 20:49:43.089319 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.088886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40293 (local address 127.0.0.1:58562)
0929 20:49:43.089069 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:43.089158 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 20:49:43.090368 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.090056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40293 (local address 127.0.0.1:58568)
0929 20:49:43.090190 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:43.090260 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 20:49:43.090567 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40293: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
I20260929 20:49:43.158759 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } attempt: 2
I20260929 20:49:43.159861 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.159430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40293 (local address 127.0.0.1:58570)
0929 20:49:43.159625 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:43.159711 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 20:49:43.160720 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.160431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40293 (local address 127.0.0.1:58584)
0929 20:49:43.160557 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:43.160621 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 20:49:43.160912 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40293: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
W20260929 20:49:43.160880 13973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:43.161165 13973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:43.161259 13973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:43.168438 13973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:49:43.168541 13973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:43.168592 13973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:43.168648 13973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:49:43.168689 13973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:49:43.178324 13973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43069,127.0.97.61:40947,127.0.97.60:40293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:43.179596 13973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:43.180827 13973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:43.191730 13979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:43.191779 13981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:43.191799 13978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:43.192646 13973 server_base.cc:1034] running on GCE node
I20260929 20:49:43.193315 13973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:43.194630 13973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:43.195889 13973 hybrid_clock.cc:648] HybridClock initialized: now 1790714983195782 us; error 46 us; skew 500 ppm
I20260929 20:49:43.196298 13973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:43 dist-test-slave-blv7 krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:43.199369 13973 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 20:49:43.200841 13973 webserver.cc:460] Webserver started at http://127.0.97.60:43555/ using document root <none> and password file <none>
I20260929 20:49:43.201581 13973 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:43.201718 13973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:49:43.202019 13973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:49:43.205106 13973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-2/data/instance:
uuid: "ba4a70cf266f4a0aba6df21d212d070f"
format_stamp: "Formatted at 2026-09-29 20:49:43 on dist-test-slave-blv7"
server_key: "546ad8632787fe2e819984190f786fc8"
server_key_iv: "1bb353c529c5fa8a3eb5748639d2aa5a"
server_key_version: "encryptionkey@0"
I20260929 20:49:43.205924 13973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-2/wal/instance:
uuid: "ba4a70cf266f4a0aba6df21d212d070f"
format_stamp: "Formatted at 2026-09-29 20:49:43 on dist-test-slave-blv7"
server_key: "546ad8632787fe2e819984190f786fc8"
server_key_iv: "1bb353c529c5fa8a3eb5748639d2aa5a"
server_key_version: "encryptionkey@0"
I20260929 20:49:43.209699 13966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } attempt: 4
I20260929 20:49:43.210878 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.210355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40293 (local address 127.0.0.1:58600)
0929 20:49:43.210602 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:43.210685 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 20:49:43.211594 13973 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 20:49:43.211901 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.211535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40293 (local address 127.0.0.1:58606)
0929 20:49:43.211690 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:43.211775 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 20:49:43.212118 13966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40293: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40293: connect: Connection refused (error 111)
I20260929 20:49:43.215147 13988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:43.216470 13973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:49:43.216609 13973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-2/wal
uuid: "ba4a70cf266f4a0aba6df21d212d070f"
format_stamp: "Formatted at 2026-09-29 20:49:43 on dist-test-slave-blv7"
server_key: "546ad8632787fe2e819984190f786fc8"
server_key_iv: "1bb353c529c5fa8a3eb5748639d2aa5a"
server_key_version: "encryptionkey@0"
I20260929 20:49:43.216771 13973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:43.242528 13973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:43.246649 13973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:43.246907 13973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:43.264040 13973 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:40293
I20260929 20:49:43.264076 14039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:40293 every 8 connection(s)
I20260929 20:49:43.265316 13973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-2/data/info.pb
I20260929 20:49:43.267089 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } attempt: 3
I20260929 20:49:43.267534 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 13973
I20260929 20:49:43.267714 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790714786636870-388-0/minicluster-data/master-2/wal/instance
I20260929 20:49:43.268239 388 external_mini_cluster.cc:1428] Setting key 7e40f2490dadd404abb3ae33255245e2
I20260929 20:49:43.269629 14040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:49:43.279132 14040 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:49:43 dist-test-slave-blv7 krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 20:49:43.290773 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.267677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40293 (local address 127.0.0.1:58620)
0929 20:49:43.268010 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:43.268029 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:43.268169 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:43.272774 (+ 4605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:43.272780 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:43.273271 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:49:43.273461 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:43.273473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.276461 (+ 2988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.276466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:43.277839 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:43.277856 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.279201 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.279208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:43.279402 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:43.280033 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:49:43.280070 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:49:43.282963 (+ 2893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:49:43.287631 (+ 4668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:43.287637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:43.287641 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:43.288144 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:43.288601 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:43.288604 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:43.288607 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:43.288764 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:43.289762 (+ 998us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:49:43.289767 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:49:43.289935 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 20:49:43.290078 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 20:49:43.290353 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260929 20:49:43.291890 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.268672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58620 (local address 127.0.97.60:40293)
0929 20:49:43.269520 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:43.269535 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:43.269586 (+ 51us) server_negotiation.cc:407] Connection header received
0929 20:49:43.271828 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:43.271855 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:43.272335 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:43.273700 (+ 1365us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:43.274235 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.276272 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.278020 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.278994 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.279763 (+ 769us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:43.283402 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:43.283413 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:43.283432 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:43.283479 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:43.287474 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.288268 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.288276 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.288284 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.288477 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.288885 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.288891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.288895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.289617 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:43.289788 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:43.290341 (+ 553us) server_negotiation.cc:299] Negotiation successful
0929 20:49:43.290734 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":341,"threads_started":1}
Sep 29 20:49:43 dist-test-slave-blv7 krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 29 20:49:43 dist-test-slave-blv7 krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 20:49:43.298892 13892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a: Bootstrap starting.
I20260929 20:49:43.301659 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.282162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.0.97.62:43069)
0929 20:49:43.282356 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:43.282362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:43.285533 (+ 3171us) server_negotiation.cc:407] Connection header received
0929 20:49:43.285686 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:43.285689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:43.285744 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:43.285816 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:43.287162 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.288114 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.290502 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.291521 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.291630 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:43.295163 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:43.295173 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:43.295177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:43.295211 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:43.298627 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.299462 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.299468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.299471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.299672 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.300075 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.300081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.300084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.300427 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:43.300614 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:43.301094 (+ 480us) server_negotiation.cc:299] Negotiation successful
0929 20:49:43.301295 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:49:43.301707 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.282210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43069 (local address 127.0.0.1:56498)
0929 20:49:43.285395 (+ 3185us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:43.285440 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:43.285583 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:43.285892 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:43.285899 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:43.286638 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:49:43.287005 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:43.287045 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.288646 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.288651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:43.290343 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:43.290356 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.291906 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.291911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:43.292087 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:43.292673 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:49:43.292689 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:49:43.294941 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:49:43.298758 (+ 3817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:43.298769 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:43.298777 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:43.299351 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:43.299793 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:43.299798 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:43.299802 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:43.299952 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:43.300574 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:49:43.300582 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:49:43.300771 (+ 189us) client_negotiation.cc:769] Sending connection context
0929 20:49:43.300954 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 20:49:43.301294 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2963,"mutex_wait_us":50,"thread_start_us":2128,"threads_started":1}
I20260929 20:49:43.302618 13892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a: Neither blocks nor log segments found. Creating new log.
I20260929 20:49:43.303406 14040 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:49:43.303834 13892 log.cc:824] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:49:43.303957 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.269503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58626 (local address 127.0.97.60:40293)
0929 20:49:43.284992 (+ 15489us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:43.284997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:43.285371 (+ 374us) server_negotiation.cc:407] Connection header received
0929 20:49:43.285427 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:43.285432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:43.285532 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:43.287144 (+ 1612us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:43.288739 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.289847 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.291762 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.293370 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.293500 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:43.297743 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:43.297748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:43.297751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:43.297778 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:43.300462 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.301267 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.301272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.301275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.301394 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.301830 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.301835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.301839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.302191 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:43.302420 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:43.303477 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0929 20:49:43.303658 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":15342,"thread_start_us":15161,"threads_started":1}
I20260929 20:49:43.307597 13892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a: No bootstrap required, opened a new log
Sep 29 20:49:43 dist-test-slave-blv7 krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 29 20:49:43 dist-test-slave-blv7 krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:49:43.315958 13892 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 } } peers { permanent_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } } peers { permanent_uuid: "ba4a70cf266f4a0aba6df21d212d070f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } }
I20260929 20:49:43.316579 13892 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:49:43.316763 13892 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea4043d82d7e4eba8b8f9c5ee153c09a, State: Initialized, Role: FOLLOWER
I20260929 20:49:43.317925 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.304417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38958 (local address 127.0.97.61:40947)
0929 20:49:43.304581 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:43.304586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:43.304600 (+ 14us) server_negotiation.cc:407] Connection header received
0929 20:49:43.304775 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:43.304781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:43.304844 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:43.304921 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:43.305664 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.306555 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.308116 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.308525 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.308641 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:43.311919 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:43.311926 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:43.311931 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:43.311979 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:43.315116 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.315753 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.315778 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.315780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.315952 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.316397 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.316402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.316406 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.316716 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:43.316840 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:43.317369 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 20:49:43.317584 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 20:49:43.318957 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.304054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40947 (local address 127.0.0.1:38958)
0929 20:49:43.304384 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:43.304401 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:43.304655 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:43.304960 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:43.304965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:43.305343 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:49:43.305507 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:43.305519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.306728 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.306734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:43.307990 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:43.308002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.308668 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.308674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:43.308871 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:43.309372 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:49:43.309392 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:49:43.311709 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:49:43.315235 (+ 3526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:43.315241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:43.315246 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:43.315648 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:43.316104 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:43.316108 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:43.316111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:43.316271 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:43.316939 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:49:43.316944 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:49:43.317105 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 20:49:43.318364 (+ 1259us) client_negotiation.cc:240] Negotiation successful
0929 20:49:43.318633 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227}
I20260929 20:49:43.320207 14040 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:49:43.320432 13892 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 } } peers { permanent_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } } peers { permanent_uuid: "ba4a70cf266f4a0aba6df21d212d070f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } }
I20260929 20:49:43.323943 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.306042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56514 (local address 127.0.97.62:43069)
0929 20:49:43.306183 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:43.306187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:43.306204 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:49:43.306246 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:43.306250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:43.306309 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:43.306401 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:43.307858 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.308977 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.310947 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.311667 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.311793 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:43.316641 (+ 4848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:43.316647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:43.316652 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:43.316691 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:43.319706 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.320665 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.320671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.320674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.321383 (+ 709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.321883 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.321889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.321893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.322255 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:43.322749 (+ 494us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:43.323349 (+ 600us) server_negotiation.cc:299] Negotiation successful
0929 20:49:43.323594 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 20:49:43 dist-test-slave-blv7 krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 20:49:43.331005 14047 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 } } peers { permanent_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } } peers { permanent_uuid: "ba4a70cf266f4a0aba6df21d212d070f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } } }
I20260929 20:49:43.331533 14047 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:49:43.332405 13892 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:49:43.334914 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.321009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58642 (local address 127.0.97.60:40293)
0929 20:49:43.321218 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:43.321224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:43.321373 (+ 149us) server_negotiation.cc:407] Connection header received
0929 20:49:43.321532 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:43.321537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:43.321589 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:43.321657 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:43.322518 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.323569 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.325185 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.325768 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.325851 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:43.329508 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:43.329514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:43.329519 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:43.329551 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:43.332354 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.332921 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.332926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.332929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.333055 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.333404 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.333408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.333411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.333764 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:43.333888 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:43.334386 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 20:49:43.334603 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:49:43.335943 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.320847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40293 (local address 127.0.0.1:58642)
0929 20:49:43.321307 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:43.321324 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:43.321442 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:43.321803 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:43.321808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:43.322234 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:49:43.322398 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:43.322408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.323701 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.323706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:43.325016 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:43.325030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.326065 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.326071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:43.326248 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:43.326800 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:49:43.326821 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:49:43.329338 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:49:43.332469 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:43.332474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:43.332479 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:43.332818 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:43.333166 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:43.333171 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:43.333173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:43.333310 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:43.333968 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:49:43.333972 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:49:43.334115 (+ 143us) client_negotiation.cc:769] Sending connection context
0929 20:49:43.335332 (+ 1217us) client_negotiation.cc:240] Negotiation successful
0929 20:49:43.335617 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288}
Sep 29 20:49:43 dist-test-slave-blv7 krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 20:49:43.342211 14040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f: Bootstrap starting.
I20260929 20:49:43.346221 14040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f: Neither blocks nor log segments found. Creating new log.
I20260929 20:49:43.346755 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.326939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38970 (local address 127.0.97.61:40947)
0929 20:49:43.327331 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:43.327338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:43.327557 (+ 219us) server_negotiation.cc:407] Connection header received
0929 20:49:43.327800 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:43.327806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:43.327886 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:43.327954 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:43.329813 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.331417 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.333128 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.333766 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.333877 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:43.339693 (+ 5816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:43.339700 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:43.339705 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:43.339749 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:43.342978 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.343867 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.343873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.343877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.344031 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.344485 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.344489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.344492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.344837 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:43.345102 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:43.346203 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0929 20:49:43.346416 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265}
I20260929 20:49:43.348645 14040 log.cc:824] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:49:43.351171 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:49:43.352030 14040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f: No bootstrap required, opened a new log
W20260929 20:49:43.357586 14058 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:49:43.357712 14058 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:49:43.348217 13825 master_runner.cc:186] Error getting master registration for 127.0.97.61:40947: OK, instance_id { permanent_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" instance_seqno: 1790714982790050 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:49:43.358633 13825 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:49:43.359375 14040 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 } } peers { permanent_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } } peers { permanent_uuid: "ba4a70cf266f4a0aba6df21d212d070f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } }
I20260929 20:49:43.359704 14040 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:49:43.359836 14040 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba4a70cf266f4a0aba6df21d212d070f, State: Initialized, Role: FOLLOWER
I20260929 20:49:43.360522 14040 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 } } peers { permanent_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } } peers { permanent_uuid: "ba4a70cf266f4a0aba6df21d212d070f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } }
I20260929 20:49:43.379619 14066 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 } } peers { permanent_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } } peers { permanent_uuid: "ba4a70cf266f4a0aba6df21d212d070f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } } }
I20260929 20:49:43.381071 14040 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:49:43.381955 14066 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:49:43.388424 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.360741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58650 (local address 127.0.97.60:40293)
0929 20:49:43.360892 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:43.360898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:43.369569 (+ 8671us) server_negotiation.cc:407] Connection header received
0929 20:49:43.369882 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:43.369889 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:43.369984 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:43.370094 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:43.372578 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.373781 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.375967 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.376673 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.376771 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:43.380574 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:43.380581 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:43.380585 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:43.380644 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:43.384112 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.385049 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.385056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.385060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.385219 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.385669 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.385675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.385679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.386082 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:43.386335 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:43.387818 (+ 1483us) server_negotiation.cc:299] Negotiation successful
0929 20:49:43.388080 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:49:43.396778 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.360405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56518 (local address 127.0.97.62:43069)
0929 20:49:43.360591 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:43.360597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:43.380630 (+ 20033us) server_negotiation.cc:407] Connection header received
0929 20:49:43.380886 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:43.380891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:43.380970 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:43.381041 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:43.382850 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.383984 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.385956 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.386651 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.386746 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:43.389960 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:43.389966 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:43.389971 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:43.390010 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:43.393470 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.394355 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.394361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.394364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.394532 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.394970 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.394975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.394978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.395356 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:43.395588 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:43.396188 (+ 600us) server_negotiation.cc:299] Negotiation successful
0929 20:49:43.396416 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:49:43.399575 14066 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:49:43.399837 14066 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 } } peers { permanent_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } } peers { permanent_uuid: "ba4a70cf266f4a0aba6df21d212d070f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } }
I20260929 20:49:43.401450 14066 leader_election.cc:288] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea4043d82d7e4eba8b8f9c5ee153c09a (127.0.97.62:43069), b559d96ad22d4c83bd3ae6051d8ecff0 (127.0.97.61:40947)
I20260929 20:49:43.402604 13867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba4a70cf266f4a0aba6df21d212d070f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a" is_pre_election: true
I20260929 20:49:43.403301 13867 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba4a70cf266f4a0aba6df21d212d070f in term 0.
I20260929 20:49:43.403875 13992 leader_election.cc:302] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba4a70cf266f4a0aba6df21d212d070f, ea4043d82d7e4eba8b8f9c5ee153c09a; no voters:
I20260929 20:49:43.404199 14066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:49:43.404345 14066 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:49:43.404439 14066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:49:43.405017 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.380814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38982 (local address 127.0.97.61:40947)
0929 20:49:43.380986 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:43.380992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:43.388416 (+ 7424us) server_negotiation.cc:407] Connection header received
0929 20:49:43.389256 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:43.389261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:43.389344 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:43.389414 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:43.391058 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.392221 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.394222 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.394903 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.394996 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:43.398531 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:43.398537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:43.398541 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:43.398583 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:43.401931 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.402682 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.402688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.402692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.402850 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.403323 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.403328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.403331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.403673 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:43.403897 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:43.404451 (+ 554us) server_negotiation.cc:299] Negotiation successful
0929 20:49:43.404683 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:49:43.407706 14066 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 } } peers { permanent_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } } peers { permanent_uuid: "ba4a70cf266f4a0aba6df21d212d070f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } }
I20260929 20:49:43.408365 14066 leader_election.cc:288] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [CANDIDATE]: Term 1 election: Requested vote from peers ea4043d82d7e4eba8b8f9c5ee153c09a (127.0.97.62:43069), b559d96ad22d4c83bd3ae6051d8ecff0 (127.0.97.61:40947)
I20260929 20:49:43.409090 13867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba4a70cf266f4a0aba6df21d212d070f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a"
I20260929 20:49:43.409301 13867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:49:43.412449 13867 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba4a70cf266f4a0aba6df21d212d070f in term 1.
I20260929 20:49:43.412902 13992 leader_election.cc:302] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba4a70cf266f4a0aba6df21d212d070f, ea4043d82d7e4eba8b8f9c5ee153c09a; no voters:
I20260929 20:49:43.413545 14066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:49:43.424147 14066 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [term 1 LEADER]: Becoming Leader. State: Replica: ba4a70cf266f4a0aba6df21d212d070f, State: Running, Role: LEADER
I20260929 20:49:43.424826 14066 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 } } peers { permanent_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } } peers { permanent_uuid: "ba4a70cf266f4a0aba6df21d212d070f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } }
I20260929 20:49:43.433670 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba4a70cf266f4a0aba6df21d212d070f. Latest consensus state: current_term: 1 leader_uuid: "ba4a70cf266f4a0aba6df21d212d070f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 } } peers { permanent_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } } peers { permanent_uuid: "ba4a70cf266f4a0aba6df21d212d070f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } } }
I20260929 20:49:43.434572 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [sys.catalog]: This master's current role is: LEADER
I20260929 20:49:43.435915 14079 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:49:43.450718 14079 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:49:43.457799 14083 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 20:49:43.473047 13867 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a [term 1 FOLLOWER]: Refusing update from remote peer ba4a70cf266f4a0aba6df21d212d070f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:49:43.474020 14066 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:49:43.463884 13973 master_runner.cc:186] Error getting master registration for 127.0.97.61:40947: OK, instance_id { permanent_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" instance_seqno: 1790714982790050 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:49:43.479183 13973 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 20:49:43.499065 13994 tablet.cc:2369] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:49:43.506199 14047 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba4a70cf266f4a0aba6df21d212d070f. Latest consensus state: current_term: 1 leader_uuid: "ba4a70cf266f4a0aba6df21d212d070f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 } } peers { permanent_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } } peers { permanent_uuid: "ba4a70cf266f4a0aba6df21d212d070f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } } }
I20260929 20:49:43.506495 14047 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:49:43.510869 14085 mvcc.cc:205] Tried to move back new op lower bound from 7334768572294762496 to 7334768572113637376. Current Snapshot: MvccSnapshot[applied={T|T < 7334768572294762496}]
I20260929 20:49:43.511289 13966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } attempt: 5
I20260929 20:49:43.515597 14066 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ba4a70cf266f4a0aba6df21d212d070f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 } } peers { permanent_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } } peers { permanent_uuid: "ba4a70cf266f4a0aba6df21d212d070f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } } }
I20260929 20:49:43.515887 14066 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [sys.catalog]: This master's current role is: LEADER
I20260929 20:49:43.517717 14079 catalog_manager.cc:1347] Generated new cluster ID: 0d5976cee2784899954f81d37349a939
I20260929 20:49:43.517830 14079 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 20:49:43 dist-test-slave-blv7 krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 20:49:43.540361 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.511949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40293 (local address 127.0.0.1:58654)
0929 20:49:43.512300 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:43.512318 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:43.512427 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:43.513410 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:43.513415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:43.513913 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:49:43.514110 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:43.514122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.515740 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.515746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:43.517185 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:43.517200 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.518573 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.518580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:43.518771 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:43.522500 (+ 3729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:49:43.522692 (+ 192us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:49:43.527012 (+ 4320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:49:43.534080 (+ 7068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:43.534087 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:43.534092 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:43.535491 (+ 1399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:43.536282 (+ 791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:43.536287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:43.536290 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:43.536599 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:43.537748 (+ 1149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:49:43.537754 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:49:43.538144 (+ 390us) client_negotiation.cc:769] Sending connection context
0929 20:49:43.538367 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 20:49:43.538885 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227}
I20260929 20:49:43.541769 14047 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ba4a70cf266f4a0aba6df21d212d070f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 } } peers { permanent_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } } peers { permanent_uuid: "ba4a70cf266f4a0aba6df21d212d070f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } } }
I20260929 20:49:43.542014 14047 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:49:43.545230 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:43.512775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58654 (local address 127.0.97.60:40293)
0929 20:49:43.512941 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:43.512947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:43.512964 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:49:43.513019 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:43.513024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:43.513099 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:43.513192 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:43.514310 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.515530 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.517392 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:43.518133 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:43.518246 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:43.527349 (+ 9103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:43.527357 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:43.527362 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:43.527415 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:43.533787 (+ 6372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.535785 (+ 1998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.535791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.535795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.536111 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:43.536825 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:43.536830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:43.536834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:43.537529 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:43.539436 (+ 1907us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:43.539961 (+ 525us) server_negotiation.cc:299] Negotiation successful
0929 20:49:43.543358 (+ 3397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:49:43.551566 13966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b559d96ad22d4c83bd3ae6051d8ecff0: Bootstrap starting.
I20260929 20:49:43.558823 13966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b559d96ad22d4c83bd3ae6051d8ecff0: Neither blocks nor log segments found. Creating new log.
I20260929 20:49:43.562119 13966 log.cc:824] T 00000000000000000000000000000000 P b559d96ad22d4c83bd3ae6051d8ecff0: Log is configured to *not* fsync() on all Append() calls
I20260929 20:49:43.564373 14079 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:49:43.568648 14079 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:49:43.576328 13966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b559d96ad22d4c83bd3ae6051d8ecff0: No bootstrap required, opened a new log
I20260929 20:49:43.593497 13966 raft_consensus.cc:348] T 00000000000000000000000000000000 P b559d96ad22d4c83bd3ae6051d8ecff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 } } peers { permanent_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } } peers { permanent_uuid: "ba4a70cf266f4a0aba6df21d212d070f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } }
I20260929 20:49:43.593976 13966 raft_consensus.cc:374] T 00000000000000000000000000000000 P b559d96ad22d4c83bd3ae6051d8ecff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:49:43.594094 13966 raft_consensus.cc:729] T 00000000000000000000000000000000 P b559d96ad22d4c83bd3ae6051d8ecff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b559d96ad22d4c83bd3ae6051d8ecff0, State: Initialized, Role: FOLLOWER
I20260929 20:49:43.595120 13966 consensus_queue.cc:260] T 00000000000000000000000000000000 P b559d96ad22d4c83bd3ae6051d8ecff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 } } peers { permanent_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } } peers { permanent_uuid: "ba4a70cf266f4a0aba6df21d212d070f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } }
I20260929 20:49:43.595264 14079 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f: Generated new TSK 0
I20260929 20:49:43.597688 14079 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:49:43.602138 14090 sys_catalog.cc:455] T 00000000000000000000000000000000 P b559d96ad22d4c83bd3ae6051d8ecff0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 } } peers { permanent_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } } peers { permanent_uuid: "ba4a70cf266f4a0aba6df21d212d070f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } } }
I20260929 20:49:43.602569 14090 sys_catalog.cc:458] T 00000000000000000000000000000000 P b559d96ad22d4c83bd3ae6051d8ecff0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:49:43.604453 13966 sys_catalog.cc:564] T 00000000000000000000000000000000 P b559d96ad22d4c83bd3ae6051d8ecff0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:49:43.604516 13941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba4a70cf266f4a0aba6df21d212d070f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" is_pre_election: true
I20260929 20:49:43.605130 13941 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b559d96ad22d4c83bd3ae6051d8ecff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba4a70cf266f4a0aba6df21d212d070f in term 0.
I20260929 20:49:43.606387 13939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b559d96ad22d4c83bd3ae6051d8ecff0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:49:43.609926 13939 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b559d96ad22d4c83bd3ae6051d8ecff0 [term 1 FOLLOWER]: Refusing update from remote peer ba4a70cf266f4a0aba6df21d212d070f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:49:43.610569 13940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba4a70cf266f4a0aba6df21d212d070f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0"
I20260929 20:49:43.611367 14071 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:49:43.614970 13940 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b559d96ad22d4c83bd3ae6051d8ecff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba4a70cf266f4a0aba6df21d212d070f in term 1.
I20260929 20:49:43.691197 14090 sys_catalog.cc:455] T 00000000000000000000000000000000 P b559d96ad22d4c83bd3ae6051d8ecff0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba4a70cf266f4a0aba6df21d212d070f. Latest consensus state: current_term: 1 leader_uuid: "ba4a70cf266f4a0aba6df21d212d070f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 } } peers { permanent_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } } peers { permanent_uuid: "ba4a70cf266f4a0aba6df21d212d070f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } } }
I20260929 20:49:43.691531 14090 sys_catalog.cc:458] T 00000000000000000000000000000000 P b559d96ad22d4c83bd3ae6051d8ecff0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:49:43.691979 14098 sys_catalog.cc:455] T 00000000000000000000000000000000 P b559d96ad22d4c83bd3ae6051d8ecff0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ba4a70cf266f4a0aba6df21d212d070f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 } } peers { permanent_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } } peers { permanent_uuid: "ba4a70cf266f4a0aba6df21d212d070f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } } }
I20260929 20:49:43.692408 14098 sys_catalog.cc:458] T 00000000000000000000000000000000 P b559d96ad22d4c83bd3ae6051d8ecff0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:49:43.717835 14110 catalog_manager.cc:1259] Loaded cluster ID: 0d5976cee2784899954f81d37349a939
I20260929 20:49:43.718014 14110 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b559d96ad22d4c83bd3ae6051d8ecff0: loading cluster ID for follower catalog manager: success
I20260929 20:49:43.721845 14110 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b559d96ad22d4c83bd3ae6051d8ecff0: acquiring CA information for follower catalog manager: success
I20260929 20:49:43.724467 14110 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b559d96ad22d4c83bd3ae6051d8ecff0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:49:44.036396 14071 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ba4a70cf266f4a0aba6df21d212d070f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea4043d82d7e4eba8b8f9c5ee153c09a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43069 } } peers { permanent_uuid: "b559d96ad22d4c83bd3ae6051d8ecff0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40947 } } peers { permanent_uuid: "ba4a70cf266f4a0aba6df21d212d070f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40293 } } }
I20260929 20:49:44.036751 14071 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba4a70cf266f4a0aba6df21d212d070f [sys.catalog]: This master's current role is: LEADER
I20260929 20:49:44.360176 14058 catalog_manager.cc:1259] Loaded cluster ID: 0d5976cee2784899954f81d37349a939
I20260929 20:49:44.360272 14058 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a: loading cluster ID for follower catalog manager: success
I20260929 20:49:44.362637 14058 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a: acquiring CA information for follower catalog manager: success
I20260929 20:49:44.364353 14058 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ea4043d82d7e4eba8b8f9c5ee153c09a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:49:44.745213 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 13825
I20260929 20:49:44.746397 13887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:49:44.853611 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 13825
I20260929 20:49:44.870150 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 13899
I20260929 20:49:44.871197 13961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:49:44.974249 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 13899
I20260929 20:49:44.990345 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 13973
I20260929 20:49:44.991415 14035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:49:45.095434 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 13973
2026-09-29T20:49:45Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4409 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:49:45 dist-test-slave-blv7 krb5kdc[14118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:49:45 dist-test-slave-blv7 krb5kdc[14118](info): set up 2 sockets
Sep 29 20:49:45 dist-test-slave-blv7 krb5kdc[14118](info): commencing operation
krb5kdc: starting...
W20260929 20:49:46.103272 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:49:46 dist-test-slave-blv7 krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:49:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:49:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:49:46.308552 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40383
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:49:46.622833 14131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:46.623188 14131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:46.623342 14131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:46.630769 14131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:49:46.630904 14131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:46.630995 14131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:46.631117 14131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:49:46.631182 14131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:49:46.641228 14131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:46.642592 14131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:46.643970 14131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:46.655942 14136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:46.655884 14137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:46.656191 14139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:46.656594 14131 server_base.cc:1034] running on GCE node
I20260929 20:49:46.657334 14131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:46.658713 14131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:46.659922 14131 hybrid_clock.cc:648] HybridClock initialized: now 1790714986659874 us; error 29 us; skew 500 ppm
I20260929 20:49:46.660291 14131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:46 dist-test-slave-blv7 krb5kdc[14118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:46.663421 14131 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:49:46.664903 14131 webserver.cc:460] Webserver started at http://127.0.97.62:38929/ using document root <none> and password file <none>
I20260929 20:49:46.665578 14131 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:46.665678 14131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:49:46.666008 14131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:49:46.668891 14131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "817b3d87937444dbb1d52b931b9ea1df"
format_stamp: "Formatted at 2026-09-29 20:49:46 on dist-test-slave-blv7"
server_key: "e226e4667686125acbb3d21090bf7ca3"
server_key_iv: "d4943094ad9703f8bc3ae2c723ecded3"
server_key_version: "encryptionkey@0"
I20260929 20:49:46.669610 14131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "817b3d87937444dbb1d52b931b9ea1df"
format_stamp: "Formatted at 2026-09-29 20:49:46 on dist-test-slave-blv7"
server_key: "e226e4667686125acbb3d21090bf7ca3"
server_key_iv: "d4943094ad9703f8bc3ae2c723ecded3"
server_key_version: "encryptionkey@0"
I20260929 20:49:46.674523 14131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:49:46.677933 14146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:46.679266 14131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:49:46.679414 14131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "817b3d87937444dbb1d52b931b9ea1df"
format_stamp: "Formatted at 2026-09-29 20:49:46 on dist-test-slave-blv7"
server_key: "e226e4667686125acbb3d21090bf7ca3"
server_key_iv: "d4943094ad9703f8bc3ae2c723ecded3"
server_key_version: "encryptionkey@0"
I20260929 20:49:46.679584 14131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:46.720961 14131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:46.724882 14131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:46.725109 14131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:46.741834 14131 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40383
I20260929 20:49:46.741863 14197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40383 every 8 connection(s)
I20260929 20:49:46.743171 14131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:49:46.746868 14198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:49:46.747397 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 14131
I20260929 20:49:46.747597 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:49:46.748191 388 external_mini_cluster.cc:1428] Setting key c80cce4c5cac3870e199f83aba955689
I20260929 20:49:46.759963 14198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df: Bootstrap starting.
Sep 29 20:49:46 dist-test-slave-blv7 krb5kdc[14118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:49:46.763458 14198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df: Neither blocks nor log segments found. Creating new log.
I20260929 20:49:46.764478 14198 log.cc:824] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df: Log is configured to *not* fsync() on all Append() calls
I20260929 20:49:46.767501 14198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df: No bootstrap required, opened a new log
I20260929 20:49:46.771127 14201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:46.749884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50576 (local address 127.0.97.62:40383)
0929 20:49:46.750612 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:46.750623 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:46.750661 (+ 38us) server_negotiation.cc:407] Connection header received
0929 20:49:46.752538 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:46.752560 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:46.752943 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:46.753285 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:46.755067 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:46.756359 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:46.758151 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:46.758781 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:46.758899 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:46.763430 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:46.763437 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:46.763455 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:46.763490 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:46.766733 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:46.767420 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:46.767428 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:46.767435 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:46.767651 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:46.767987 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:46.767991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:46.767994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:46.768666 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:46.768961 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:46.769355 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 20:49:46.769792 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":313,"threads_started":1}
I20260929 20:49:46.774753 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:49:46.776137 14198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "817b3d87937444dbb1d52b931b9ea1df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } }
I20260929 20:49:46.776463 14198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:49:46.776553 14198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 817b3d87937444dbb1d52b931b9ea1df, State: Initialized, Role: FOLLOWER
I20260929 20:49:46.777156 14198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "817b3d87937444dbb1d52b931b9ea1df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } }
I20260929 20:49:46.777383 14198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:49:46.777508 14198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:49:46.777674 14198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:49:46.779815 14198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "817b3d87937444dbb1d52b931b9ea1df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } }
I20260929 20:49:46.780246 14198 leader_election.cc:302] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 817b3d87937444dbb1d52b931b9ea1df; no voters:
I20260929 20:49:46.780730 14198 leader_election.cc:288] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:49:46.780850 14203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:49:46.781483 14203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df [term 1 LEADER]: Becoming Leader. State: Replica: 817b3d87937444dbb1d52b931b9ea1df, State: Running, Role: LEADER
I20260929 20:49:46.782008 14203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "817b3d87937444dbb1d52b931b9ea1df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } }
I20260929 20:49:46.782634 14198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:49:46.784686 14204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "817b3d87937444dbb1d52b931b9ea1df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "817b3d87937444dbb1d52b931b9ea1df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } } }
I20260929 20:49:46.784592 14205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 817b3d87937444dbb1d52b931b9ea1df. Latest consensus state: current_term: 1 leader_uuid: "817b3d87937444dbb1d52b931b9ea1df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "817b3d87937444dbb1d52b931b9ea1df" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40383 } } }
I20260929 20:49:46.784931 14204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df [sys.catalog]: This master's current role is: LEADER
I20260929 20:49:46.784932 14205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df [sys.catalog]: This master's current role is: LEADER
I20260929 20:49:46.786404 14209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:49:46.790934 14209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:49:46.798233 14209 catalog_manager.cc:1347] Generated new cluster ID: 2147f86295ba447a89c4cb0a9a6c073d
I20260929 20:49:46.798333 14209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:49:46.823278 14209 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:49:46.825222 14209 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:49:46.844149 14209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 817b3d87937444dbb1d52b931b9ea1df: Generated new TSK 0
I20260929 20:49:46.845176 14209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:49:46.886075 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 14131
I20260929 20:49:46.887207 14193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:49:46.982113 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 14131
2026-09-29T20:49:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1882 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:49:47 dist-test-slave-blv7 krb5kdc[14226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:49:47 dist-test-slave-blv7 krb5kdc[14226](info): set up 2 sockets
Sep 29 20:49:47 dist-test-slave-blv7 krb5kdc[14226](info): commencing operation
krb5kdc: starting...
W20260929 20:49:47.987267 388 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:49:48 dist-test-slave-blv7 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:49:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:49:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:49:48.198083 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44053
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:49:48.508539 14239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:48.508826 14239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:48.508952 14239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:48.516036 14239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:49:48.516147 14239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:48.516222 14239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:48.516300 14239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:49:48.516356 14239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:49:48.525854 14239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:48.527102 14239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:48.528358 14239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:48.539343 14244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:48.539287 14245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:48.539322 14247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:48.540393 14239 server_base.cc:1034] running on GCE node
I20260929 20:49:48.541107 14239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:48.542447 14239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:48.543642 14239 hybrid_clock.cc:648] HybridClock initialized: now 1790714988543589 us; error 39 us; skew 500 ppm
I20260929 20:49:48.543960 14239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:48 dist-test-slave-blv7 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:48.550441 14239 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:49:48.551874 14239 webserver.cc:460] Webserver started at http://127.0.97.62:33225/ using document root <none> and password file <none>
I20260929 20:49:48.552516 14239 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:48.552634 14239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:49:48.552902 14239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:49:48.555783 14239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "6c1bff5661b649058bc06d5ff580cbd8"
format_stamp: "Formatted at 2026-09-29 20:49:48 on dist-test-slave-blv7"
server_key: "2dd1f5b6c21581dfd5c5536e4b521788"
server_key_iv: "95b7a7e178569a52962e6f04004a3495"
server_key_version: "encryptionkey@0"
I20260929 20:49:48.556511 14239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "6c1bff5661b649058bc06d5ff580cbd8"
format_stamp: "Formatted at 2026-09-29 20:49:48 on dist-test-slave-blv7"
server_key: "2dd1f5b6c21581dfd5c5536e4b521788"
server_key_iv: "95b7a7e178569a52962e6f04004a3495"
server_key_version: "encryptionkey@0"
I20260929 20:49:48.561353 14239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:49:48.564585 14254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:48.565784 14239 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 20:49:48.565929 14239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "6c1bff5661b649058bc06d5ff580cbd8"
format_stamp: "Formatted at 2026-09-29 20:49:48 on dist-test-slave-blv7"
server_key: "2dd1f5b6c21581dfd5c5536e4b521788"
server_key_iv: "95b7a7e178569a52962e6f04004a3495"
server_key_version: "encryptionkey@0"
I20260929 20:49:48.566107 14239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:48.596122 14239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:48.600306 14239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:48.600564 14239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:48.617224 14239 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44053
I20260929 20:49:48.617245 14305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44053 every 8 connection(s)
I20260929 20:49:48.618523 14239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:49:48.622105 14306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:49:48.627307 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 14239
I20260929 20:49:48.627491 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:49:48.628077 388 external_mini_cluster.cc:1428] Setting key 07fbdf9ce83fabf5ffef794461783da2
I20260929 20:49:48.632612 14306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8: Bootstrap starting.
I20260929 20:49:48.636252 14306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8: Neither blocks nor log segments found. Creating new log.
I20260929 20:49:48.637475 14306 log.cc:824] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8: Log is configured to *not* fsync() on all Append() calls
I20260929 20:49:48.640529 14306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8: No bootstrap required, opened a new log
Sep 29 20:49:48 dist-test-slave-blv7 krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:49:48.649376 14306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1bff5661b649058bc06d5ff580cbd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44053 } }
I20260929 20:49:48.649824 14306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:49:48.649928 14306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c1bff5661b649058bc06d5ff580cbd8, State: Initialized, Role: FOLLOWER
I20260929 20:49:48.650645 14306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1bff5661b649058bc06d5ff580cbd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44053 } }
I20260929 20:49:48.650947 14306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:49:48.651116 14306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:49:48.651291 14306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:49:48.652539 14309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:48.630123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48552 (local address 127.0.97.62:44053)
0929 20:49:48.630960 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:48.630974 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:48.631363 (+ 389us) server_negotiation.cc:407] Connection header received
0929 20:49:48.633680 (+ 2317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:48.633707 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:48.634160 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:48.634614 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:48.635814 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:48.637776 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:48.639347 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:48.640324 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:48.640489 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:48.644221 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:48.644236 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:48.644254 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:48.644309 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:48.648148 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:48.648793 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:48.648802 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:48.648811 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:48.649007 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:48.649315 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:48.649320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:48.649323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:48.650023 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:48.650310 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:48.650692 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 20:49:48.651167 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":350,"threads_started":1}
I20260929 20:49:48.654063 14306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1bff5661b649058bc06d5ff580cbd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44053 } }
I20260929 20:49:48.654632 14306 leader_election.cc:302] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c1bff5661b649058bc06d5ff580cbd8; no voters:
I20260929 20:49:48.655352 14306 leader_election.cc:288] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:49:48.655467 14311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:49:48.656279 14311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 1 LEADER]: Becoming Leader. State: Replica: 6c1bff5661b649058bc06d5ff580cbd8, State: Running, Role: LEADER
I20260929 20:49:48.656760 14311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1bff5661b649058bc06d5ff580cbd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44053 } }
I20260929 20:49:48.657260 14306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:49:48.659411 14313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c1bff5661b649058bc06d5ff580cbd8. Latest consensus state: current_term: 1 leader_uuid: "6c1bff5661b649058bc06d5ff580cbd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1bff5661b649058bc06d5ff580cbd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44053 } } }
I20260929 20:49:48.659750 14313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [sys.catalog]: This master's current role is: LEADER
I20260929 20:49:48.661134 14312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c1bff5661b649058bc06d5ff580cbd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1bff5661b649058bc06d5ff580cbd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44053 } } }
I20260929 20:49:48.661698 14312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [sys.catalog]: This master's current role is: LEADER
I20260929 20:49:48.661960 14318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:49:48.667672 14318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:49:48.675660 14318 catalog_manager.cc:1347] Generated new cluster ID: a78f75b8fcdc43fa8b5bc988852ae480
I20260929 20:49:48.675769 14318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:49:48.696811 14318 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:49:48.698715 14318 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:49:48.712793 14318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8: Generated new TSK 0
I20260929 20:49:48.713830 14318 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:49:48.763731 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44053
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:49:49.076507 14332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:49.076769 14332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:49.076865 14332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:49.083837 14332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:49.083940 14332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:49.084055 14332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:49:49.093361 14332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:49.094483 14332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:49.095676 14332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:49.107817 14337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:49.107848 14338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:49.107780 14340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:49.108453 14332 server_base.cc:1034] running on GCE node
I20260929 20:49:49.109037 14332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:49.109970 14332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:49.111133 14332 hybrid_clock.cc:648] HybridClock initialized: now 1790714989111085 us; error 34 us; skew 500 ppm
I20260929 20:49:49.111433 14332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:49 dist-test-slave-blv7 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:49.114455 14332 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:49:49.115761 14332 webserver.cc:460] Webserver started at http://127.0.97.1:42609/ using document root <none> and password file <none>
I20260929 20:49:49.116384 14332 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:49.116500 14332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:49:49.116794 14332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:49:49.119689 14332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "5583b1ffe1694d959621cb3540c7057d"
format_stamp: "Formatted at 2026-09-29 20:49:49 on dist-test-slave-blv7"
server_key: "6c0daeb80ea2f4122524fcacb6de0052"
server_key_iv: "7d16113fe13386f1bab159f5400c0ffa"
server_key_version: "encryptionkey@0"
I20260929 20:49:49.120405 14332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5583b1ffe1694d959621cb3540c7057d"
format_stamp: "Formatted at 2026-09-29 20:49:49 on dist-test-slave-blv7"
server_key: "6c0daeb80ea2f4122524fcacb6de0052"
server_key_iv: "7d16113fe13386f1bab159f5400c0ffa"
server_key_version: "encryptionkey@0"
I20260929 20:49:49.125150 14332 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 20:49:49.128530 14347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:49.129886 14332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:49:49.130039 14332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "5583b1ffe1694d959621cb3540c7057d"
format_stamp: "Formatted at 2026-09-29 20:49:49 on dist-test-slave-blv7"
server_key: "6c0daeb80ea2f4122524fcacb6de0052"
server_key_iv: "7d16113fe13386f1bab159f5400c0ffa"
server_key_version: "encryptionkey@0"
I20260929 20:49:49.130220 14332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:49.146379 14332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:49.150127 14332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:49.150377 14332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:49.151324 14332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:49:49.152592 14332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:49:49.152684 14332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:49.152806 14332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:49:49.152863 14332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:49.183039 14332 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37283
I20260929 20:49:49.183064 14459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37283 every 8 connection(s)
I20260929 20:49:49.184381 14332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:49:49.193508 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 14332
I20260929 20:49:49.193740 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:49:49.194634 388 external_mini_cluster.cc:1428] Setting key 462784922488de380f0ed6869cf42a78
Sep 29 20:49:49 dist-test-slave-blv7 krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:49:49.204635 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:49.186848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45637 (local address 127.0.97.62:44053)
0929 20:49:49.187330 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:49.187335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:49.188254 (+ 919us) server_negotiation.cc:407] Connection header received
0929 20:49:49.190072 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:49.190077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:49.190142 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:49.190226 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:49.191689 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:49.192406 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:49.194521 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:49.194951 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:49.195084 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:49.199170 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:49.199175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:49.199178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:49.199211 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:49.201591 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:49.202432 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:49.202435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:49.202437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:49.202553 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:49.203001 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:49.203004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:49.203006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:49.203301 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:49.203428 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:49.204110 (+ 682us) server_negotiation.cc:299] Negotiation successful
0929 20:49:49.204347 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":260,"threads_started":1}
I20260929 20:49:49.205468 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:49.187383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.1:45637)
0929 20:49:49.188087 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:49.188133 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:49.189799 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:49.190436 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:49.190444 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:49.190934 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:49:49.191550 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:49.191568 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:49.192552 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:49.192556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:49.194333 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:49.194349 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:49.195194 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:49.195203 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:49.195465 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:49.196185 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:49:49.196209 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:49:49.198918 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:49:49.201733 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:49.201742 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:49.201759 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:49.202315 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:49.202653 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:49.202657 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:49.202661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:49.202903 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:49.203435 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:49:49.203444 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:49:49.203759 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 20:49:49.203998 (+ 239us) client_negotiation.cc:240] Negotiation successful
0929 20:49:49.204371 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":257,"threads_started":1}
I20260929 20:49:49.207072 14460 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44053
I20260929 20:49:49.207399 14460 heartbeater.cc:464] Registering TS with master...
I20260929 20:49:49.208117 14460 heartbeater.cc:511] Master 127.0.97.62:44053 requested a full tablet report, sending...
I20260929 20:49:49.210373 14271 ts_manager.cc:194] Registered new tserver with Master: 5583b1ffe1694d959621cb3540c7057d (127.0.97.1:37283)
I20260929 20:49:49.212438 14271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45637
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:49:49.242183 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44053
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:49:49.554857 14466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:49.555135 14466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:49.555233 14466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:49.562649 14466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:49.562752 14466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:49.562872 14466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:49:49.572280 14466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:49.573450 14466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:49.574669 14466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:49.586949 14472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:49.586990 14474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:49.586925 14471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:49.587697 14466 server_base.cc:1034] running on GCE node
I20260929 20:49:49.588286 14466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:49.589143 14466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:49.590346 14466 hybrid_clock.cc:648] HybridClock initialized: now 1790714989590294 us; error 42 us; skew 500 ppm
I20260929 20:49:49.590648 14466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:49 dist-test-slave-blv7 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:49.593655 14466 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:49:49.594973 14466 webserver.cc:460] Webserver started at http://127.0.97.2:38367/ using document root <none> and password file <none>
I20260929 20:49:49.595610 14466 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:49.595718 14466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:49:49.595991 14466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:49:49.598768 14466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "65fab7ae9a23408a994ea10f870f1d6c"
format_stamp: "Formatted at 2026-09-29 20:49:49 on dist-test-slave-blv7"
server_key: "1424bcd628ad6b37c2039c5ef19a7578"
server_key_iv: "53ac5d2d6462b72ea1689dffc96b3cd2"
server_key_version: "encryptionkey@0"
I20260929 20:49:49.599534 14466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "65fab7ae9a23408a994ea10f870f1d6c"
format_stamp: "Formatted at 2026-09-29 20:49:49 on dist-test-slave-blv7"
server_key: "1424bcd628ad6b37c2039c5ef19a7578"
server_key_iv: "53ac5d2d6462b72ea1689dffc96b3cd2"
server_key_version: "encryptionkey@0"
I20260929 20:49:49.604415 14466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 20:49:49.607628 14481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:49.608825 14466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:49:49.608992 14466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "65fab7ae9a23408a994ea10f870f1d6c"
format_stamp: "Formatted at 2026-09-29 20:49:49 on dist-test-slave-blv7"
server_key: "1424bcd628ad6b37c2039c5ef19a7578"
server_key_iv: "53ac5d2d6462b72ea1689dffc96b3cd2"
server_key_version: "encryptionkey@0"
I20260929 20:49:49.609174 14466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:49.642669 14466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:49.646708 14466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:49.646960 14466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:49.647909 14466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:49:49.649145 14466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:49:49.649219 14466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:49.649330 14466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:49:49.649386 14466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:49.677574 14466 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35321
I20260929 20:49:49.677620 14593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35321 every 8 connection(s)
I20260929 20:49:49.678764 14466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:49:49.681424 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 14466
I20260929 20:49:49.681604 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:49:49.682127 388 external_mini_cluster.cc:1428] Setting key 3e0e96fc0287411de829b674dbb05f52
Sep 29 20:49:49 dist-test-slave-blv7 krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:49:49.697593 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:49.681194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42963 (local address 127.0.97.62:44053)
0929 20:49:49.681359 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:49.681363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:49.682707 (+ 1344us) server_negotiation.cc:407] Connection header received
0929 20:49:49.684715 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:49.684720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:49.684802 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:49.684888 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:49.686361 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:49.687103 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:49.688604 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:49.689007 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:49.689112 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:49.692426 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:49.692432 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:49.692435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:49.692466 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:49.694782 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:49.695476 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:49.695482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:49.695485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:49.695611 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:49.696006 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:49.696009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:49.696011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:49.696254 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:49.696380 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:49.697045 (+ 665us) server_negotiation.cc:299] Negotiation successful
0929 20:49:49.697268 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:49:49.698550 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:49.681809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.2:42963)
0929 20:49:49.682571 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:49.682606 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:49.684466 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:49.685128 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:49.685136 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:49.685608 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:49:49.686209 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:49.686227 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:49.687249 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:49.687252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:49.688476 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:49.688487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:49.689207 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:49.689215 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:49.689427 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:49.690008 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:49:49.690030 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:49:49.692229 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:49:49.694915 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:49.694923 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:49.694940 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:49.695373 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:49.695708 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:49.695713 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:49.695717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:49.695898 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:49.696368 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:49:49.696375 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:49:49.696695 (+ 320us) client_negotiation.cc:769] Sending connection context
0929 20:49:49.696919 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 20:49:49.697418 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":328,"threads_started":1}
I20260929 20:49:49.700019 14594 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44053
I20260929 20:49:49.700351 14594 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:49:49.701210 14594 heartbeater.cc:511] Master 127.0.97.62:44053 requested a full tablet report, sending...
I20260929 20:49:49.702698 14271 ts_manager.cc:194] Registered new tserver with Master: 65fab7ae9a23408a994ea10f870f1d6c (127.0.97.2:35321)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 20:49:49.703943 14271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:49:49.729638 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44053
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:49:50.036481 14599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:50.036760 14599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:50.036852 14599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:50.044086 14599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:50.044198 14599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:50.044315 14599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:49:50.053874 14599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:50.055104 14599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:50.056347 14599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:50.068717 14607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:50.068747 14605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:50.068950 14604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:50.069730 14599 server_base.cc:1034] running on GCE node
I20260929 20:49:50.070334 14599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:50.071209 14599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:50.072403 14599 hybrid_clock.cc:648] HybridClock initialized: now 1790714990072359 us; error 38 us; skew 500 ppm
I20260929 20:49:50.072683 14599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:50 dist-test-slave-blv7 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:50.075547 14599 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:49:50.076879 14599 webserver.cc:460] Webserver started at http://127.0.97.3:40453/ using document root <none> and password file <none>
I20260929 20:49:50.077515 14599 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:50.077623 14599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:49:50.077888 14599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:49:50.080680 14599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "b45ac457da574e91a068aa2bb2127d5a"
format_stamp: "Formatted at 2026-09-29 20:49:50 on dist-test-slave-blv7"
server_key: "98e30136355852f3f8b6fa8fecb807a3"
server_key_iv: "c0e719525c87e7633023209bab206075"
server_key_version: "encryptionkey@0"
I20260929 20:49:50.081416 14599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b45ac457da574e91a068aa2bb2127d5a"
format_stamp: "Formatted at 2026-09-29 20:49:50 on dist-test-slave-blv7"
server_key: "98e30136355852f3f8b6fa8fecb807a3"
server_key_iv: "c0e719525c87e7633023209bab206075"
server_key_version: "encryptionkey@0"
I20260929 20:49:50.086210 14599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260929 20:49:50.089474 14614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:50.090703 14599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 20:49:50.090835 14599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "b45ac457da574e91a068aa2bb2127d5a"
format_stamp: "Formatted at 2026-09-29 20:49:50 on dist-test-slave-blv7"
server_key: "98e30136355852f3f8b6fa8fecb807a3"
server_key_iv: "c0e719525c87e7633023209bab206075"
server_key_version: "encryptionkey@0"
I20260929 20:49:50.091007 14599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:50.110302 14599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:50.114348 14599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:50.114586 14599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:50.115501 14599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:49:50.116709 14599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:49:50.116786 14599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:50.116904 14599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:49:50.116964 14599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:50.145305 14599 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33365
I20260929 20:49:50.145370 14726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33365 every 8 connection(s)
I20260929 20:49:50.146515 14599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:49:50.147675 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 14599
I20260929 20:49:50.147850 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:49:50.148379 388 external_mini_cluster.cc:1428] Setting key b2c92b1c1f7278d9d29cd0a5c6922d89
Sep 29 20:49:50 dist-test-slave-blv7 krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:49:50.165994 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:50.149330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43047 (local address 127.0.97.62:44053)
0929 20:49:50.149522 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:50.149528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:50.150805 (+ 1277us) server_negotiation.cc:407] Connection header received
0929 20:49:50.152740 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:50.152745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:50.152820 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:50.152903 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:50.154329 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.155054 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.156536 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.156986 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.157100 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:50.160285 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:50.160290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:50.160293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:50.160326 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:50.163151 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:50.163829 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:50.163835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:50.163839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:50.163970 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:50.164359 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:50.164365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:50.164369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:50.164655 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:50.164783 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:50.165444 (+ 661us) server_negotiation.cc:299] Negotiation successful
0929 20:49:50.165652 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:49:50.166911 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:50.149903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.3:43047)
0929 20:49:50.150671 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:50.150708 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:50.152487 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:50.153157 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:50.153166 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:50.153648 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:49:50.154174 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:50.154191 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.155202 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.155208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:50.156403 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:50.156415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.157217 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.157224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:50.157441 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:50.158016 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:49:50.158041 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:49:50.160076 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:49:50.163265 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:50.163274 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:50.163291 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:50.163711 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:50.164070 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:50.164075 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:50.164079 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:50.164257 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:50.164774 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:49:50.164782 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:49:50.165098 (+ 316us) client_negotiation.cc:769] Sending connection context
0929 20:49:50.165331 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 20:49:50.165854 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":317,"threads_started":1}
I20260929 20:49:50.168130 14727 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44053
I20260929 20:49:50.168378 14727 heartbeater.cc:464] Registering TS with master...
I20260929 20:49:50.169047 14727 heartbeater.cc:511] Master 127.0.97.62:44053 requested a full tablet report, sending...
I20260929 20:49:50.170413 14271 ts_manager.cc:194] Registered new tserver with Master: b45ac457da574e91a068aa2bb2127d5a (127.0.97.3:33365)
I20260929 20:49:50.171267 14271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43047
I20260929 20:49:50.175546 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:49:50.191185 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:50.179732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48562 (local address 127.0.97.62:44053)
0929 20:49:50.179886 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:50.179890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:50.180346 (+ 456us) server_negotiation.cc:407] Connection header received
0929 20:49:50.180472 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:50.180475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:50.180546 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:50.180638 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:50.181834 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.182533 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.183865 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.184275 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.184380 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:50.186543 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:50.186548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:50.186551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:50.186579 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:50.188790 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:50.189393 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:50.189397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:50.189399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:50.189505 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:50.189831 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:50.189835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:50.189839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:50.190139 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:50.190359 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:50.190680 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 20:49:50.190889 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 20:49:50.194546 14271 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:49:50.201164 14271 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:49:50.215634 14460 heartbeater.cc:503] Master 127.0.97.62:44053 was elected leader, sending a full tablet report...
I20260929 20:49:50.222834 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:50.212963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40090 (local address 127.0.97.3:33365)
0929 20:49:50.215812 (+ 2849us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:50.215822 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:50.215857 (+ 35us) server_negotiation.cc:407] Connection header received
0929 20:49:50.215960 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:50.215969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:50.216248 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:50.216453 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:49:50.217140 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.218476 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.220202 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.221636 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.221773 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:50.222015 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:50.222313 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 20:49:50.222484 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2666,"thread_start_us":2474,"threads_started":1}
I20260929 20:49:50.222873 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:50.212704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33365 (local address 127.0.0.1:40090)
0929 20:49:50.214071 (+ 1367us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:50.214097 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:50.214201 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:50.216451 (+ 2250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:50.216458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:50.216486 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:49:50.216913 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:50.216926 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.218603 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.218608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:50.220034 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:50.220047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.221837 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.221843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:50.222095 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:50.222137 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 20:49:50.222263 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 20:49:50.222466 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1252,"mutex_wait_us":24,"thread_start_us":1522,"threads_started":1}
I20260929 20:49:50.224908 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:50.212333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35321 (local address 127.0.0.1:58980)
0929 20:49:50.213310 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:50.213360 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:50.213583 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:50.217596 (+ 4013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:50.217601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:50.217623 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:49:50.217996 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:50.218010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.220143 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.220148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:50.222261 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:50.222276 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.224104 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.224111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:50.224280 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:50.224313 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:49:50.224406 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 20:49:50.224539 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":927,"threads_started":1}
I20260929 20:49:50.225298 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:50.214385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37283 (local address 127.0.0.1:37114)
0929 20:49:50.215074 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:50.215092 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:50.215208 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:50.218071 (+ 2863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:50.218075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:50.218110 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:49:50.218461 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:50.218472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.220504 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.220510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:50.222682 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:50.222694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.224476 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.224483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:50.224644 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:50.224680 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 20:49:50.224763 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 20:49:50.224892 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":26,"thread_start_us":421,"threads_started":1}
I20260929 20:49:50.225559 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:50.214881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37114 (local address 127.0.97.1:37283)
0929 20:49:50.216062 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:50.216070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:50.216106 (+ 36us) server_negotiation.cc:407] Connection header received
0929 20:49:50.216197 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:50.216203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:50.217841 (+ 1638us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:50.218632 (+ 791us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:49:50.218846 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.220315 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.222866 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.224287 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.224399 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:50.224625 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:50.225108 (+ 483us) server_negotiation.cc:299] Negotiation successful
0929 20:49:50.225272 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1032,"thread_start_us":1129,"threads_started":1}
I20260929 20:49:50.226395 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:50.212684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58980 (local address 127.0.97.2:35321)
0929 20:49:50.217043 (+ 4359us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:50.217050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:50.217076 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:49:50.217168 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:50.217175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:50.217385 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:50.218224 (+ 839us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:49:50.218439 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.220002 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.222441 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.223896 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.224030 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:50.224225 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:50.225919 (+ 1694us) server_negotiation.cc:299] Negotiation successful
0929 20:49:50.226078 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4229,"thread_start_us":4122,"threads_started":1}
I20260929 20:49:50.227088 14662 tablet_service.cc:1467] Processing CreateTablet for tablet 78421f8d3b504ee98b1a60e93419dfe6 (DEFAULT_TABLE table=test-table [id=fa3668dac93846f988d69a0a434c5dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:49:50.228618 14662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78421f8d3b504ee98b1a60e93419dfe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:49:50.228857 14395 tablet_service.cc:1467] Processing CreateTablet for tablet 78421f8d3b504ee98b1a60e93419dfe6 (DEFAULT_TABLE table=test-table [id=fa3668dac93846f988d69a0a434c5dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:49:50.229915 14395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78421f8d3b504ee98b1a60e93419dfe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:49:50.229781 14529 tablet_service.cc:1467] Processing CreateTablet for tablet 78421f8d3b504ee98b1a60e93419dfe6 (DEFAULT_TABLE table=test-table [id=fa3668dac93846f988d69a0a434c5dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:49:50.231145 14529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78421f8d3b504ee98b1a60e93419dfe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:49:50.241844 14745 tablet_bootstrap.cc:492] T 78421f8d3b504ee98b1a60e93419dfe6 P 5583b1ffe1694d959621cb3540c7057d: Bootstrap starting.
I20260929 20:49:50.243846 14746 tablet_bootstrap.cc:492] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a: Bootstrap starting.
I20260929 20:49:50.245558 14745 tablet_bootstrap.cc:654] T 78421f8d3b504ee98b1a60e93419dfe6 P 5583b1ffe1694d959621cb3540c7057d: Neither blocks nor log segments found. Creating new log.
I20260929 20:49:50.246803 14745 log.cc:824] T 78421f8d3b504ee98b1a60e93419dfe6 P 5583b1ffe1694d959621cb3540c7057d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:49:50.247208 14746 tablet_bootstrap.cc:654] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a: Neither blocks nor log segments found. Creating new log.
I20260929 20:49:50.247676 14747 tablet_bootstrap.cc:492] T 78421f8d3b504ee98b1a60e93419dfe6 P 65fab7ae9a23408a994ea10f870f1d6c: Bootstrap starting.
I20260929 20:49:50.248406 14746 log.cc:824] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:49:50.250792 14747 tablet_bootstrap.cc:654] T 78421f8d3b504ee98b1a60e93419dfe6 P 65fab7ae9a23408a994ea10f870f1d6c: Neither blocks nor log segments found. Creating new log.
I20260929 20:49:50.251190 14745 tablet_bootstrap.cc:492] T 78421f8d3b504ee98b1a60e93419dfe6 P 5583b1ffe1694d959621cb3540c7057d: No bootstrap required, opened a new log
I20260929 20:49:50.251307 14746 tablet_bootstrap.cc:492] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a: No bootstrap required, opened a new log
I20260929 20:49:50.251466 14745 ts_tablet_manager.cc:1397] T 78421f8d3b504ee98b1a60e93419dfe6 P 5583b1ffe1694d959621cb3540c7057d: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 20:49:50.251518 14746 ts_tablet_manager.cc:1397] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260929 20:49:50.251824 14747 log.cc:824] T 78421f8d3b504ee98b1a60e93419dfe6 P 65fab7ae9a23408a994ea10f870f1d6c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:49:50.254071 14747 tablet_bootstrap.cc:492] T 78421f8d3b504ee98b1a60e93419dfe6 P 65fab7ae9a23408a994ea10f870f1d6c: No bootstrap required, opened a new log
I20260929 20:49:50.254311 14747 ts_tablet_manager.cc:1397] T 78421f8d3b504ee98b1a60e93419dfe6 P 65fab7ae9a23408a994ea10f870f1d6c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260929 20:49:50.261415 14747 raft_consensus.cc:348] T 78421f8d3b504ee98b1a60e93419dfe6 P 65fab7ae9a23408a994ea10f870f1d6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 } } peers { permanent_uuid: "b45ac457da574e91a068aa2bb2127d5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33365 } } peers { permanent_uuid: "5583b1ffe1694d959621cb3540c7057d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37283 } }
I20260929 20:49:50.261832 14747 raft_consensus.cc:374] T 78421f8d3b504ee98b1a60e93419dfe6 P 65fab7ae9a23408a994ea10f870f1d6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:49:50.261962 14747 raft_consensus.cc:729] T 78421f8d3b504ee98b1a60e93419dfe6 P 65fab7ae9a23408a994ea10f870f1d6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65fab7ae9a23408a994ea10f870f1d6c, State: Initialized, Role: FOLLOWER
I20260929 20:49:50.261754 14745 raft_consensus.cc:348] T 78421f8d3b504ee98b1a60e93419dfe6 P 5583b1ffe1694d959621cb3540c7057d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 } } peers { permanent_uuid: "b45ac457da574e91a068aa2bb2127d5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33365 } } peers { permanent_uuid: "5583b1ffe1694d959621cb3540c7057d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37283 } }
I20260929 20:49:50.262240 14745 raft_consensus.cc:374] T 78421f8d3b504ee98b1a60e93419dfe6 P 5583b1ffe1694d959621cb3540c7057d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:49:50.262337 14745 raft_consensus.cc:729] T 78421f8d3b504ee98b1a60e93419dfe6 P 5583b1ffe1694d959621cb3540c7057d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5583b1ffe1694d959621cb3540c7057d, State: Initialized, Role: FOLLOWER
I20260929 20:49:50.262394 14746 raft_consensus.cc:348] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 } } peers { permanent_uuid: "b45ac457da574e91a068aa2bb2127d5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33365 } } peers { permanent_uuid: "5583b1ffe1694d959621cb3540c7057d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37283 } }
I20260929 20:49:50.262797 14746 raft_consensus.cc:374] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:49:50.262666 14747 consensus_queue.cc:260] T 78421f8d3b504ee98b1a60e93419dfe6 P 65fab7ae9a23408a994ea10f870f1d6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 } } peers { permanent_uuid: "b45ac457da574e91a068aa2bb2127d5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33365 } } peers { permanent_uuid: "5583b1ffe1694d959621cb3540c7057d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37283 } }
I20260929 20:49:50.262888 14746 raft_consensus.cc:729] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45ac457da574e91a068aa2bb2127d5a, State: Initialized, Role: FOLLOWER
I20260929 20:49:50.262917 14745 consensus_queue.cc:260] T 78421f8d3b504ee98b1a60e93419dfe6 P 5583b1ffe1694d959621cb3540c7057d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 } } peers { permanent_uuid: "b45ac457da574e91a068aa2bb2127d5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33365 } } peers { permanent_uuid: "5583b1ffe1694d959621cb3540c7057d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37283 } }
I20260929 20:49:50.263665 14746 consensus_queue.cc:260] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 } } peers { permanent_uuid: "b45ac457da574e91a068aa2bb2127d5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33365 } } peers { permanent_uuid: "5583b1ffe1694d959621cb3540c7057d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37283 } }
I20260929 20:49:50.264915 14727 heartbeater.cc:503] Master 127.0.97.62:44053 was elected leader, sending a full tablet report...
I20260929 20:49:50.265668 14746 ts_tablet_manager.cc:1428] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a: Time spent starting tablet: real 0.014s user 0.005s sys 0.007s
I20260929 20:49:50.267153 14745 ts_tablet_manager.cc:1428] T 78421f8d3b504ee98b1a60e93419dfe6 P 5583b1ffe1694d959621cb3540c7057d: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
I20260929 20:49:50.268699 14594 heartbeater.cc:503] Master 127.0.97.62:44053 was elected leader, sending a full tablet report...
I20260929 20:49:50.269536 14747 ts_tablet_manager.cc:1428] T 78421f8d3b504ee98b1a60e93419dfe6 P 65fab7ae9a23408a994ea10f870f1d6c: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
W20260929 20:49:50.398139 14728 tablet.cc:2369] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:49:50.411116 14753 raft_consensus.cc:482] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:49:50.411381 14753 raft_consensus.cc:504] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 } } peers { permanent_uuid: "b45ac457da574e91a068aa2bb2127d5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33365 } } peers { permanent_uuid: "5583b1ffe1694d959621cb3540c7057d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37283 } }
I20260929 20:49:50.413183 14753 leader_election.cc:288] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65fab7ae9a23408a994ea10f870f1d6c (127.0.97.2:35321), 5583b1ffe1694d959621cb3540c7057d (127.0.97.1:37283)
I20260929 20:49:50.419009 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:50.413514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33507 (local address 127.0.97.2:35321)
0929 20:49:50.413746 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:50.413752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:50.414019 (+ 267us) server_negotiation.cc:407] Connection header received
0929 20:49:50.414183 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:50.414187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:50.414251 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:50.414331 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:49:50.414788 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.415590 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.417264 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.418092 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.418226 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:50.418331 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:50.418601 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 20:49:50.418727 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:49:50.419059 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:50.413488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35321 (local address 127.0.97.3:33507)
0929 20:49:50.413855 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:50.413875 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:50.414023 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:50.414374 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:50.414379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:50.414418 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:49:50.414644 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:50.414653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.415734 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.415738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:50.417091 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:50.417111 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.418250 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.418256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:50.418448 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:50.418474 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 20:49:50.418568 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 20:49:50.418710 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":1,"spinlock_wait_cycles":17792}
I20260929 20:49:50.419267 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:50.413705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54007 (local address 127.0.97.1:37283)
0929 20:49:50.413912 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:50.413934 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:50.414461 (+ 527us) server_negotiation.cc:407] Connection header received
0929 20:49:50.414641 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:50.414645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:50.414702 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:50.414774 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:49:50.415301 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.416001 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.417615 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.418409 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.418526 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:50.418628 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:50.418895 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 20:49:50.419038 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:49:50.419474 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:50.413665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37283 (local address 127.0.97.3:54007)
0929 20:49:50.414356 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:50.414370 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:50.414510 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:50.414800 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:50.414804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:50.414827 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:49:50.415163 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:50.415173 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.416161 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.416166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:50.417463 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:50.417474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.418611 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.418618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:50.418760 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:50.418782 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 20:49:50.418859 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 20:49:50.418980 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":379,"threads_started":1}
I20260929 20:49:50.420049 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78421f8d3b504ee98b1a60e93419dfe6" candidate_uuid: "b45ac457da574e91a068aa2bb2127d5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65fab7ae9a23408a994ea10f870f1d6c" is_pre_election: true
I20260929 20:49:50.420315 14415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78421f8d3b504ee98b1a60e93419dfe6" candidate_uuid: "b45ac457da574e91a068aa2bb2127d5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5583b1ffe1694d959621cb3540c7057d" is_pre_election: true
I20260929 20:49:50.420555 14549 raft_consensus.cc:2454] T 78421f8d3b504ee98b1a60e93419dfe6 P 65fab7ae9a23408a994ea10f870f1d6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b45ac457da574e91a068aa2bb2127d5a in term 0.
I20260929 20:49:50.420822 14415 raft_consensus.cc:2454] T 78421f8d3b504ee98b1a60e93419dfe6 P 5583b1ffe1694d959621cb3540c7057d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b45ac457da574e91a068aa2bb2127d5a in term 0.
I20260929 20:49:50.421272 14618 leader_election.cc:302] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65fab7ae9a23408a994ea10f870f1d6c, b45ac457da574e91a068aa2bb2127d5a; no voters:
I20260929 20:49:50.421633 14753 raft_consensus.cc:2781] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:49:50.421774 14753 raft_consensus.cc:482] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:49:50.421867 14753 raft_consensus.cc:3037] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:49:50.424027 14753 raft_consensus.cc:504] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 } } peers { permanent_uuid: "b45ac457da574e91a068aa2bb2127d5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33365 } } peers { permanent_uuid: "5583b1ffe1694d959621cb3540c7057d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37283 } }
I20260929 20:49:50.424593 14753 leader_election.cc:288] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a [CANDIDATE]: Term 1 election: Requested vote from peers 65fab7ae9a23408a994ea10f870f1d6c (127.0.97.2:35321), 5583b1ffe1694d959621cb3540c7057d (127.0.97.1:37283)
I20260929 20:49:50.425037 14415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78421f8d3b504ee98b1a60e93419dfe6" candidate_uuid: "b45ac457da574e91a068aa2bb2127d5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5583b1ffe1694d959621cb3540c7057d"
I20260929 20:49:50.425069 14549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78421f8d3b504ee98b1a60e93419dfe6" candidate_uuid: "b45ac457da574e91a068aa2bb2127d5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65fab7ae9a23408a994ea10f870f1d6c"
I20260929 20:49:50.425209 14415 raft_consensus.cc:3037] T 78421f8d3b504ee98b1a60e93419dfe6 P 5583b1ffe1694d959621cb3540c7057d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:49:50.425241 14549 raft_consensus.cc:3037] T 78421f8d3b504ee98b1a60e93419dfe6 P 65fab7ae9a23408a994ea10f870f1d6c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:49:50.427367 14415 raft_consensus.cc:2454] T 78421f8d3b504ee98b1a60e93419dfe6 P 5583b1ffe1694d959621cb3540c7057d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b45ac457da574e91a068aa2bb2127d5a in term 1.
I20260929 20:49:50.427384 14549 raft_consensus.cc:2454] T 78421f8d3b504ee98b1a60e93419dfe6 P 65fab7ae9a23408a994ea10f870f1d6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b45ac457da574e91a068aa2bb2127d5a in term 1.
I20260929 20:49:50.427754 14618 leader_election.cc:302] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5583b1ffe1694d959621cb3540c7057d, 65fab7ae9a23408a994ea10f870f1d6c, b45ac457da574e91a068aa2bb2127d5a; no voters:
I20260929 20:49:50.428037 14753 raft_consensus.cc:2781] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:49:50.428457 14753 raft_consensus.cc:686] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a [term 1 LEADER]: Becoming Leader. State: Replica: b45ac457da574e91a068aa2bb2127d5a, State: Running, Role: LEADER
I20260929 20:49:50.428910 14753 consensus_queue.cc:237] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 } } peers { permanent_uuid: "b45ac457da574e91a068aa2bb2127d5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33365 } } peers { permanent_uuid: "5583b1ffe1694d959621cb3540c7057d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37283 } }
W20260929 20:49:50.430559 14595 tablet.cc:2369] T 78421f8d3b504ee98b1a60e93419dfe6 P 65fab7ae9a23408a994ea10f870f1d6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:49:50.433293 14270 catalog_manager.cc:5495] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a reported cstate change: term changed from 0 to 1, leader changed from <none> to b45ac457da574e91a068aa2bb2127d5a (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b45ac457da574e91a068aa2bb2127d5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b45ac457da574e91a068aa2bb2127d5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5583b1ffe1694d959621cb3540c7057d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37283 } health_report { overall_health: UNKNOWN } } }
W20260929 20:49:50.436591 14461 tablet.cc:2369] T 78421f8d3b504ee98b1a60e93419dfe6 P 5583b1ffe1694d959621cb3540c7057d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:49:50.465698 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:50.460532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40100 (local address 127.0.97.3:33365)
0929 20:49:50.460690 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:50.460695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:50.460712 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:49:50.460861 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:50.460865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:50.460962 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:50.461088 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:49:50.461484 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.462260 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.463670 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.464085 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.464199 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:50.464653 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:49:50.465154 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:49:50.465284 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:50.465383 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 20:49:50.465461 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 20:49:50.473244 14415 raft_consensus.cc:1261] T 78421f8d3b504ee98b1a60e93419dfe6 P 5583b1ffe1694d959621cb3540c7057d [term 1 FOLLOWER]: Refusing update from remote peer b45ac457da574e91a068aa2bb2127d5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:49:50.473244 14549 raft_consensus.cc:1261] T 78421f8d3b504ee98b1a60e93419dfe6 P 65fab7ae9a23408a994ea10f870f1d6c [term 1 FOLLOWER]: Refusing update from remote peer b45ac457da574e91a068aa2bb2127d5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:49:50.474123 14755 consensus_queue.cc:1035] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:49:50.474480 14753 consensus_queue.cc:1035] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5583b1ffe1694d959621cb3540c7057d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:49:50.484225 14758 mvcc.cc:205] Tried to move back new op lower bound from 7334768600969367552 to 7334768600800358400. Current Snapshot: MvccSnapshot[applied={T|T < 7334768600969367552}]
I20260929 20:49:50.486297 14760 mvcc.cc:205] Tried to move back new op lower bound from 7334768600969367552 to 7334768600800358400. Current Snapshot: MvccSnapshot[applied={T|T < 7334768600969367552}]
I20260929 20:49:50.499481 14271 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:49:50.499754 14271 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:49:50.503191 14271 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8: Sending DeleteTablet for 3 replicas of tablet 78421f8d3b504ee98b1a60e93419dfe6
I20260929 20:49:50.504292 14529 tablet_service.cc:1514] Processing DeleteTablet for tablet 78421f8d3b504ee98b1a60e93419dfe6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:49:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58980
I20260929 20:49:50.504297 14662 tablet_service.cc:1514] Processing DeleteTablet for tablet 78421f8d3b504ee98b1a60e93419dfe6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:49:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40090
I20260929 20:49:50.504645 14395 tablet_service.cc:1514] Processing DeleteTablet for tablet 78421f8d3b504ee98b1a60e93419dfe6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:49:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37114
I20260929 20:49:50.505693 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 14332
I20260929 20:49:50.532737 14770 tablet_replica.cc:330] T 78421f8d3b504ee98b1a60e93419dfe6 P 5583b1ffe1694d959621cb3540c7057d: stopping tablet replica
I20260929 20:49:50.533286 14770 raft_consensus.cc:2229] T 78421f8d3b504ee98b1a60e93419dfe6 P 5583b1ffe1694d959621cb3540c7057d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:49:50.533888 14770 raft_consensus.cc:2258] T 78421f8d3b504ee98b1a60e93419dfe6 P 5583b1ffe1694d959621cb3540c7057d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:49:50.538379 14769 tablet_replica.cc:330] T 78421f8d3b504ee98b1a60e93419dfe6 P 65fab7ae9a23408a994ea10f870f1d6c: stopping tablet replica
I20260929 20:49:50.538844 14769 raft_consensus.cc:2229] T 78421f8d3b504ee98b1a60e93419dfe6 P 65fab7ae9a23408a994ea10f870f1d6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:49:50.539363 14769 raft_consensus.cc:2258] T 78421f8d3b504ee98b1a60e93419dfe6 P 65fab7ae9a23408a994ea10f870f1d6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:49:50.539947 14770 ts_tablet_manager.cc:1905] T 78421f8d3b504ee98b1a60e93419dfe6 P 5583b1ffe1694d959621cb3540c7057d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:49:50.541412 14769 ts_tablet_manager.cc:1905] T 78421f8d3b504ee98b1a60e93419dfe6 P 65fab7ae9a23408a994ea10f870f1d6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:49:50.547518 14768 tablet_replica.cc:330] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a: stopping tablet replica
I20260929 20:49:50.547945 14768 raft_consensus.cc:2229] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:49:50.548259 14769 ts_tablet_manager.cc:1918] T 78421f8d3b504ee98b1a60e93419dfe6 P 65fab7ae9a23408a994ea10f870f1d6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:49:50.548411 14769 log.cc:1196] T 78421f8d3b504ee98b1a60e93419dfe6 P 65fab7ae9a23408a994ea10f870f1d6c: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/78421f8d3b504ee98b1a60e93419dfe6
I20260929 20:49:50.548612 14768 raft_consensus.cc:2258] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:49:50.548918 14769 ts_tablet_manager.cc:1939] T 78421f8d3b504ee98b1a60e93419dfe6 P 65fab7ae9a23408a994ea10f870f1d6c: Deleting consensus metadata
I20260929 20:49:50.550659 14258 catalog_manager.cc:4841] TS 65fab7ae9a23408a994ea10f870f1d6c (127.0.97.2:35321): tablet 78421f8d3b504ee98b1a60e93419dfe6 (table test-table [id=fa3668dac93846f988d69a0a434c5dbf]) successfully deleted
I20260929 20:49:50.551127 14768 ts_tablet_manager.cc:1905] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:49:50.552198 14770 ts_tablet_manager.cc:1918] T 78421f8d3b504ee98b1a60e93419dfe6 P 5583b1ffe1694d959621cb3540c7057d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:49:50.552358 14770 log.cc:1196] T 78421f8d3b504ee98b1a60e93419dfe6 P 5583b1ffe1694d959621cb3540c7057d: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/78421f8d3b504ee98b1a60e93419dfe6
Sep 29 20:49:50 dist-test-slave-blv7 krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:49:50.552815 14770 ts_tablet_manager.cc:1939] T 78421f8d3b504ee98b1a60e93419dfe6 P 5583b1ffe1694d959621cb3540c7057d: Deleting consensus metadata
I20260929 20:49:50.554549 14257 catalog_manager.cc:4841] TS 5583b1ffe1694d959621cb3540c7057d (127.0.97.1:37283): tablet 78421f8d3b504ee98b1a60e93419dfe6 (table test-table [id=fa3668dac93846f988d69a0a434c5dbf]) successfully deleted
I20260929 20:49:50.557197 14768 ts_tablet_manager.cc:1918] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:49:50.557340 14768 log.cc:1196] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/78421f8d3b504ee98b1a60e93419dfe6
I20260929 20:49:50.557736 14768 ts_tablet_manager.cc:1939] T 78421f8d3b504ee98b1a60e93419dfe6 P b45ac457da574e91a068aa2bb2127d5a: Deleting consensus metadata
I20260929 20:49:50.559118 14255 catalog_manager.cc:4841] TS b45ac457da574e91a068aa2bb2127d5a (127.0.97.3:33365): tablet 78421f8d3b504ee98b1a60e93419dfe6 (table test-table [id=fa3668dac93846f988d69a0a434c5dbf]) successfully deleted
I20260929 20:49:50.560986 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:50.506849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37116 (local address 127.0.97.1:37283)
0929 20:49:50.507035 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:50.507041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:50.529578 (+ 22537us) server_negotiation.cc:407] Connection header received
0929 20:49:50.530236 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:50.530242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:50.530352 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:50.530429 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:50.534084 (+ 3655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.535304 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.547392 (+ 12088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.547972 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.548090 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:50.554050 (+ 5960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:50.554065 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:50.554081 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:50.554122 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:50.557877 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:50.558733 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:50.558742 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:50.558745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:50.558908 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:50.559400 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:50.559406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:50.559410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:50.559879 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:50.560166 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:50.560392 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 20:49:50.560620 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:49:50.561610 14455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:49:50.703482 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 14332
I20260929 20:49:50.724826 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 14466
Sep 29 20:49:50 dist-test-slave-blv7 krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:49:50.739755 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:50.725683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58996 (local address 127.0.97.2:35321)
0929 20:49:50.725859 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:50.725865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:50.725885 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:49:50.726041 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:50.726047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:50.726127 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:50.726211 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:50.727434 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.728164 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.729484 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.729873 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.729980 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:50.734131 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:50.734145 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:50.734159 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:50.734195 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:50.736938 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:50.737714 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:50.737720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:50.737723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:50.737840 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:50.738208 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:50.738213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:50.738216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:50.738583 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:50.738802 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:50.739264 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 20:49:50.739426 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:49:50.740487 14589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:49:50.827728 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 14466
I20260929 20:49:50.848152 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 14599
Sep 29 20:49:50 dist-test-slave-blv7 krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:49:50.865592 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:50.848895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40108 (local address 127.0.97.3:33365)
0929 20:49:50.849062 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:50.849066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:50.849120 (+ 54us) server_negotiation.cc:407] Connection header received
0929 20:49:50.849308 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:50.849312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:50.849381 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:50.849465 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:50.851154 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.851971 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.853901 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:50.854319 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:50.854412 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:50.859633 (+ 5221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:50.859648 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:50.859664 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:50.859701 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:50.862621 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:50.863386 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:50.863397 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:50.863399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:50.863565 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:50.863932 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:50.863936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:50.863942 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:50.864288 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:50.864605 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:50.864909 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 20:49:50.865133 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:49:50.866257 14722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:49:50.987113 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 14599
I20260929 20:49:51.002754 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 14239
I20260929 20:49:51.003877 14301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:49:51.110078 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 14239
I20260929 20:49:51.126981 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44053
--webserver_interface=127.0.97.62
--webserver_port=33225
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:49:51.440801 14776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:51.441066 14776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:51.441165 14776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:51.448279 14776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:49:51.448382 14776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:51.448443 14776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:51.448500 14776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:49:51.448544 14776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:49:51.458081 14776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:51.459331 14776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:51.460619 14776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:51.482120 14783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:51.482157 14785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:51.482187 14782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:51.482930 14776 server_base.cc:1034] running on GCE node
I20260929 20:49:51.483760 14776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:51.485304 14776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:51.486506 14776 hybrid_clock.cc:648] HybridClock initialized: now 1790714991486451 us; error 43 us; skew 500 ppm
I20260929 20:49:51.486883 14776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:51 dist-test-slave-blv7 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:51.490254 14776 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:49:51.491843 14776 webserver.cc:460] Webserver started at http://127.0.97.62:33225/ using document root <none> and password file <none>
I20260929 20:49:51.492528 14776 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:51.492640 14776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:49:51.498348 14776 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:49:51.501194 14792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:51.502555 14776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:49:51.502712 14776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "6c1bff5661b649058bc06d5ff580cbd8"
format_stamp: "Formatted at 2026-09-29 20:49:48 on dist-test-slave-blv7"
server_key: "2dd1f5b6c21581dfd5c5536e4b521788"
server_key_iv: "95b7a7e178569a52962e6f04004a3495"
server_key_version: "encryptionkey@0"
I20260929 20:49:51.503794 14776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:51.520371 14776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:51.524235 14776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:51.524490 14776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:51.541312 14776 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44053
I20260929 20:49:51.541330 14843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44053 every 8 connection(s)
I20260929 20:49:51.542721 14776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:49:51.545779 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 14776
I20260929 20:49:51.546536 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:37283
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:44053
--webserver_port=42609
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:49:51.549479 14844 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:49:51.552673 14844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8: Bootstrap starting.
I20260929 20:49:51.571537 14844 log.cc:824] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8: Log is configured to *not* fsync() on all Append() calls
I20260929 20:49:51.583143 14844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:49:51.583626 14844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8: Bootstrap complete.
I20260929 20:49:51.592258 14844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1bff5661b649058bc06d5ff580cbd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44053 } }
I20260929 20:49:51.593042 14844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c1bff5661b649058bc06d5ff580cbd8, State: Initialized, Role: FOLLOWER
I20260929 20:49:51.593634 14844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1bff5661b649058bc06d5ff580cbd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44053 } }
I20260929 20:49:51.593844 14844 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:49:51.593955 14844 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:49:51.594111 14844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:49:51.597733 14844 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1bff5661b649058bc06d5ff580cbd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44053 } }
I20260929 20:49:51.598204 14844 leader_election.cc:302] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c1bff5661b649058bc06d5ff580cbd8; no voters:
I20260929 20:49:51.598765 14844 leader_election.cc:288] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 20:49:51.598892 14848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:49:51.600069 14848 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 2 LEADER]: Becoming Leader. State: Replica: 6c1bff5661b649058bc06d5ff580cbd8, State: Running, Role: LEADER
I20260929 20:49:51.600682 14848 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1bff5661b649058bc06d5ff580cbd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44053 } }
I20260929 20:49:51.600891 14844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:49:51.602986 14850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c1bff5661b649058bc06d5ff580cbd8. Latest consensus state: current_term: 2 leader_uuid: "6c1bff5661b649058bc06d5ff580cbd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1bff5661b649058bc06d5ff580cbd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44053 } } }
I20260929 20:49:51.603359 14850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [sys.catalog]: This master's current role is: LEADER
I20260929 20:49:51.611581 14849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6c1bff5661b649058bc06d5ff580cbd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1bff5661b649058bc06d5ff580cbd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44053 } } }
I20260929 20:49:51.611900 14849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [sys.catalog]: This master's current role is: LEADER
I20260929 20:49:51.612249 14853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:49:51.620338 14853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:49:51.622335 14853 catalog_manager.cc:1259] Loaded cluster ID: a78f75b8fcdc43fa8b5bc988852ae480
I20260929 20:49:51.622603 14853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:49:51.626068 14853 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 20:49:51.646021 14853 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 20:49:51.655848 14853 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:49:51.657531 14853 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8: Loaded TSK: 0
I20260929 20:49:51.659137 14853 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 20:49:51.873796 14846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:51.874064 14846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:51.874195 14846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:51.881364 14846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:51.881492 14846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:51.881636 14846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:49:51.891139 14846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:37283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42609
--tserver_master_addrs=127.0.97.62:44053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:51.892356 14846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:51.893620 14846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:51.908238 14872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:51.908262 14874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:51.909089 14846 server_base.cc:1034] running on GCE node
W20260929 20:49:51.911538 14871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:51.912328 14846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:51.913597 14846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:51.914844 14846 hybrid_clock.cc:648] HybridClock initialized: now 1790714991914778 us; error 54 us; skew 500 ppm
I20260929 20:49:51.915257 14846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:51 dist-test-slave-blv7 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:51.919446 14846 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:49:51.921310 14846 webserver.cc:460] Webserver started at http://127.0.97.1:42609/ using document root <none> and password file <none>
I20260929 20:49:51.922248 14846 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:51.922405 14846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:49:51.928464 14846 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:49:51.931308 14881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:51.932749 14846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:49:51.932891 14846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "5583b1ffe1694d959621cb3540c7057d"
format_stamp: "Formatted at 2026-09-29 20:49:49 on dist-test-slave-blv7"
server_key: "6c0daeb80ea2f4122524fcacb6de0052"
server_key_iv: "7d16113fe13386f1bab159f5400c0ffa"
server_key_version: "encryptionkey@0"
I20260929 20:49:51.933835 14846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:51.949445 14846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:51.953483 14846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:51.953732 14846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:51.954671 14846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:49:51.956169 14846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:49:51.956269 14846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:51.956370 14846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:49:51.956442 14846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:51.986474 14846 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37283
I20260929 20:49:51.986539 14993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37283 every 8 connection(s)
I20260929 20:49:51.987916 14846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:49:51.996894 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 14846
I20260929 20:49:51.997442 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35321
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:44053
--webserver_port=38367
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:49:51 dist-test-slave-blv7 krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:49:52.010077 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:51.990836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.1:39763)
0929 20:49:51.991603 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:51.991643 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:51.993265 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:51.994579 (+ 1314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:51.994588 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:51.995228 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:49:51.995753 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:51.995769 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:51.997402 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:51.997406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:51.998643 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:51.998655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:51.999646 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:51.999655 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:51.999903 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:52.000519 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:49:52.000548 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:49:52.002615 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:49:52.006308 (+ 3693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:52.006315 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:52.006326 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:52.006752 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:52.007184 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:52.007187 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:52.007189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:52.007383 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:52.008193 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:49:52.008199 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:49:52.008522 (+ 323us) client_negotiation.cc:769] Sending connection context
0929 20:49:52.008758 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 20:49:52.009131 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":309,"threads_started":1}
I20260929 20:49:52.010439 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:51.990628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39763 (local address 127.0.97.62:44053)
0929 20:49:51.991384 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:51.991400 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:51.991758 (+ 358us) server_negotiation.cc:407] Connection header received
0929 20:49:51.993773 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:51.993786 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:51.994108 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:51.994417 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:51.995951 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:51.997259 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:51.998773 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:51.999435 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:51.999615 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:52.002857 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:52.002864 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:52.002878 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:52.002915 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:52.006155 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:52.006863 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:52.006870 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:52.006878 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:52.007086 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:52.007478 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:52.007482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:52.007486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:52.008061 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:52.008242 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:52.008981 (+ 739us) server_negotiation.cc:299] Negotiation successful
0929 20:49:52.009374 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":340,"threads_started":1}
I20260929 20:49:52.013110 14994 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44053
I20260929 20:49:52.013440 14994 heartbeater.cc:464] Registering TS with master...
I20260929 20:49:52.014330 14994 heartbeater.cc:511] Master 127.0.97.62:44053 requested a full tablet report, sending...
I20260929 20:49:52.016532 14809 ts_manager.cc:194] Registered new tserver with Master: 5583b1ffe1694d959621cb3540c7057d (127.0.97.1:37283)
I20260929 20:49:52.017946 14809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39763
W20260929 20:49:52.307971 14998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:52.308238 14998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:52.308404 14998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:52.315573 14998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:52.315690 14998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:52.315846 14998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:49:52.325636 14998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=38367
--tserver_master_addrs=127.0.97.62:44053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:52.326893 14998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:52.328176 14998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:52.340562 15003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:52.340600 15004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:52.340509 15006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:52.341455 14998 server_base.cc:1034] running on GCE node
I20260929 20:49:52.342012 14998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:52.342895 14998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:52.344089 14998 hybrid_clock.cc:648] HybridClock initialized: now 1790714992344018 us; error 57 us; skew 500 ppm
I20260929 20:49:52.344389 14998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:52 dist-test-slave-blv7 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:52.347204 14998 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:49:52.348474 14998 webserver.cc:460] Webserver started at http://127.0.97.2:38367/ using document root <none> and password file <none>
I20260929 20:49:52.349088 14998 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:52.349220 14998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:49:52.354236 14998 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 20:49:52.356784 15013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:52.358013 14998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 20:49:52.358166 14998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "65fab7ae9a23408a994ea10f870f1d6c"
format_stamp: "Formatted at 2026-09-29 20:49:49 on dist-test-slave-blv7"
server_key: "1424bcd628ad6b37c2039c5ef19a7578"
server_key_iv: "53ac5d2d6462b72ea1689dffc96b3cd2"
server_key_version: "encryptionkey@0"
I20260929 20:49:52.359125 14998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:52.385052 14998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:52.388935 14998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:52.389196 14998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:52.390156 14998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:49:52.391479 14998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:49:52.391573 14998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:52.391682 14998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:49:52.391749 14998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:52.420456 14998 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35321
I20260929 20:49:52.420522 15125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35321 every 8 connection(s)
I20260929 20:49:52.421893 14998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:49:52.428544 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 14998
I20260929 20:49:52.429078 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:33365
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:44053
--webserver_port=40453
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:49:52 dist-test-slave-blv7 krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:49:52.441521 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:52.424710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36397 (local address 127.0.97.62:44053)
0929 20:49:52.424879 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:52.424883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:52.426029 (+ 1146us) server_negotiation.cc:407] Connection header received
0929 20:49:52.427853 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:52.427858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:52.427939 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:52.428023 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:52.429557 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.430228 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.431931 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.432345 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.432453 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:52.435837 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:52.435843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:52.435848 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:52.435885 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:52.438702 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:52.439427 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:52.439432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:52.439435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:52.439571 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:52.439911 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:52.439915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:52.439918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:52.440239 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:52.440377 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:52.440986 (+ 609us) server_negotiation.cc:299] Negotiation successful
0929 20:49:52.441213 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:49:52.442759 15128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:52.425192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.2:36397)
0929 20:49:52.425880 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:52.425915 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:52.427590 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:52.428264 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:52.428272 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:52.428801 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:49:52.429402 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:52.429422 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.430384 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.430388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:52.431802 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:52.431814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.432546 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.432553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:52.432784 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:52.433373 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:49:52.433395 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:49:52.435637 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:49:52.438829 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:52.438835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:52.438848 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:52.439314 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:52.439654 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:52.439657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:52.439659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:52.439822 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:52.440341 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:49:52.440346 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:49:52.440631 (+ 285us) client_negotiation.cc:769] Sending connection context
0929 20:49:52.440852 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 20:49:52.441341 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":261,"threads_started":1}
I20260929 20:49:52.445890 15126 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44053
I20260929 20:49:52.446234 15126 heartbeater.cc:464] Registering TS with master...
I20260929 20:49:52.447132 15126 heartbeater.cc:511] Master 127.0.97.62:44053 requested a full tablet report, sending...
I20260929 20:49:52.448702 14809 ts_manager.cc:194] Registered new tserver with Master: 65fab7ae9a23408a994ea10f870f1d6c (127.0.97.2:35321)
I20260929 20:49:52.449806 14809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36397
W20260929 20:49:52.745124 15129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:52.745389 15129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:52.745520 15129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:52.752617 15129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:52.752740 15129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:52.752911 15129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:49:52.762562 15129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:33365
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40453
--tserver_master_addrs=127.0.97.62:44053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:52.763795 15129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:52.765053 15129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:52.777176 15137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:52.777205 15135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:52.777262 15134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:52.783476 15129 server_base.cc:1034] running on GCE node
I20260929 20:49:52.784060 15129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:52.784981 15129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:52.786185 15129 hybrid_clock.cc:648] HybridClock initialized: now 1790714992786134 us; error 54 us; skew 500 ppm
I20260929 20:49:52.786494 15129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:52 dist-test-slave-blv7 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:52.789528 15129 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:49:52.790948 15129 webserver.cc:460] Webserver started at http://127.0.97.3:40453/ using document root <none> and password file <none>
I20260929 20:49:52.791643 15129 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:52.791759 15129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:49:52.796775 15129 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 20:49:52.799256 15144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:52.800499 15129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:49:52.800647 15129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "b45ac457da574e91a068aa2bb2127d5a"
format_stamp: "Formatted at 2026-09-29 20:49:50 on dist-test-slave-blv7"
server_key: "98e30136355852f3f8b6fa8fecb807a3"
server_key_iv: "c0e719525c87e7633023209bab206075"
server_key_version: "encryptionkey@0"
I20260929 20:49:52.801599 15129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:52.819456 15129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:52.824065 15129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:52.824345 15129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:52.825372 15129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:49:52.826756 15129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:49:52.826861 15129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:52.826977 15129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:49:52.827082 15129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 20:49:52.857560 15129 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33365
I20260929 20:49:52.857630 15256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33365 every 8 connection(s)
I20260929 20:49:52.859066 15129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:49:52.860642 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 15129
Sep 29 20:49:52 dist-test-slave-blv7 krb5kdc[14226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:49:52.877552 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:52.861799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48572 (local address 127.0.97.62:44053)
0929 20:49:52.863917 (+ 2118us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:52.863922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:52.864259 (+ 337us) server_negotiation.cc:407] Connection header received
0929 20:49:52.864425 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:52.864431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:52.864501 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:52.864589 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:52.866193 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.866909 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.868761 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.869159 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.869259 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:52.872020 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:52.872025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:52.872028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:52.872057 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:52.874935 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:52.875708 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:52.875714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:52.875721 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:52.875907 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:52.876253 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:52.876259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:52.876262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:52.876674 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:52.876923 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:52.877119 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 20:49:52.877278 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2004,"thread_start_us":1906,"threads_started":1}
I20260929 20:49:52.880043 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:52.861545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49599 (local address 127.0.97.62:44053)
0929 20:49:52.861837 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:52.861845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:52.863544 (+ 1699us) server_negotiation.cc:407] Connection header received
0929 20:49:52.865623 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:52.865627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:52.865697 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:52.865805 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:52.867254 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.868385 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.869916 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.870510 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.870625 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:52.873853 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:52.873859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:52.873864 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:52.873909 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:52.876949 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:52.877854 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:52.877859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:52.877862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:52.877988 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:52.878405 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:52.878408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:52.878410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:52.878664 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:52.878812 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:52.879539 (+ 727us) server_negotiation.cc:299] Negotiation successful
0929 20:49:52.879728 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 20:49:52.881071 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:52.862166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.3:49599)
0929 20:49:52.863279 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:52.863396 (+ 117us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:52.865354 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:52.866034 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:52.866042 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:52.866508 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:49:52.867107 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:52.867127 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.868531 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.868534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:52.869784 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:52.869799 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.870719 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.870724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:52.870915 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:52.871491 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:49:52.871513 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:49:52.873650 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:49:52.877078 (+ 3428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:52.877086 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:52.877119 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:52.877731 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:52.878089 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:49:52.878101 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:49:52.878105 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:49:52.878306 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:49:52.878799 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:49:52.878808 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:49:52.879188 (+ 380us) client_negotiation.cc:769] Sending connection context
0929 20:49:52.879408 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 20:49:52.880071 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":372,"threads_started":1}
I20260929 20:49:52.882274 15257 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44053
I20260929 20:49:52.882542 15257 heartbeater.cc:464] Registering TS with master...
I20260929 20:49:52.883235 15257 heartbeater.cc:511] Master 127.0.97.62:44053 requested a full tablet report, sending...
I20260929 20:49:52.884478 14809 ts_manager.cc:194] Registered new tserver with Master: b45ac457da574e91a068aa2bb2127d5a (127.0.97.3:33365)
I20260929 20:49:52.885214 14809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49599
I20260929 20:49:52.891803 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:49:52.897634 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:52.892584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48580 (local address 127.0.97.62:44053)
0929 20:49:52.892735 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:52.892739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:52.893144 (+ 405us) server_negotiation.cc:407] Connection header received
0929 20:49:52.893307 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:52.893311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:52.893432 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:52.893557 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:49:52.893888 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.894588 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.895953 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.896323 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.896408 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:52.896784 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:49:52.897238 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:49:52.897335 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:52.897404 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 20:49:52.897462 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:49:52.898401 14809 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:49:52.905634 14809 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:49:52.926662 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:52.917690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37132 (local address 127.0.97.1:37283)
0929 20:49:52.918265 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:52.918274 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:52.918895 (+ 621us) server_negotiation.cc:407] Connection header received
0929 20:49:52.918996 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:52.919003 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:52.919293 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:52.919472 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:49:52.920487 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.921954 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.923986 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.925417 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.925532 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:52.925720 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:52.926136 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 20:49:52.926314 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":853,"threads_started":1}
I20260929 20:49:52.926702 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:52.916735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37283 (local address 127.0.0.1:37132)
0929 20:49:52.918653 (+ 1918us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:52.918670 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:52.918787 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:52.919718 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:52.919723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:52.919756 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:49:52.920291 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:52.920309 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.922066 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.922071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:52.923769 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:52.923784 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.925649 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.925656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:52.925881 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:52.925934 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 20:49:52.926067 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 20:49:52.926314 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1786,"mutex_wait_us":61,"thread_start_us":270,"threads_started":1}
I20260929 20:49:52.926801 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:52.916726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59004 (local address 127.0.97.2:35321)
0929 20:49:52.918931 (+ 2205us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:52.918939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:52.918972 (+ 33us) server_negotiation.cc:407] Connection header received
0929 20:49:52.919087 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:52.919095 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:52.919275 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:52.919437 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:49:52.920139 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.921497 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.924374 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.925593 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.925710 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:52.925934 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:52.926306 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 20:49:52.926467 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2040,"thread_start_us":1989,"threads_started":1}
I20260929 20:49:52.927423 15268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:52.916430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35321 (local address 127.0.0.1:59004)
0929 20:49:52.917416 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:52.917469 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:52.917637 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:52.919459 (+ 1822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:52.919466 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:52.919493 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:49:52.919935 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:52.919952 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.921631 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.921637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:52.924202 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:52.924215 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.925894 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.925900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:52.926048 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:52.926079 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:49:52.926168 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 20:49:52.927117 (+ 949us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":858,"thread_start_us":1968,"threads_started":1}
I20260929 20:49:52.928653 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:52.918292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.0.97.3:33365)
0929 20:49:52.919051 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:52.919058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:52.920508 (+ 1450us) server_negotiation.cc:407] Connection header received
0929 20:49:52.920720 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:52.920728 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:52.920935 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:52.921098 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:49:52.921902 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.923474 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.925992 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.927336 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.927458 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:52.927678 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:52.928183 (+ 505us) server_negotiation.cc:299] Negotiation successful
0929 20:49:52.928338 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":569,"threads_started":1}
I20260929 20:49:52.930500 14929 tablet_service.cc:1467] Processing CreateTablet for tablet 493096d0edae42e1958204676bc8cb06 (DEFAULT_TABLE table=test-table [id=c6a8e0663f7343cea3aea3fb83a54da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:49:52.931522 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:52.917293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33365 (local address 127.0.0.1:40110)
0929 20:49:52.920338 (+ 3045us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:52.920354 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:52.920547 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:52.921338 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:52.921343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:52.921366 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:49:52.921711 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:52.921721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.923645 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.923650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:52.925822 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:52.925837 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.927650 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.927658 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:52.927822 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:52.927853 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:49:52.927948 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 20:49:52.931226 (+ 3278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2908,"thread_start_us":2588,"threads_started":1}
I20260929 20:49:52.932049 14929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493096d0edae42e1958204676bc8cb06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:49:52.933425 15061 tablet_service.cc:1467] Processing CreateTablet for tablet 493096d0edae42e1958204676bc8cb06 (DEFAULT_TABLE table=test-table [id=c6a8e0663f7343cea3aea3fb83a54da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:49:52.934836 15061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493096d0edae42e1958204676bc8cb06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:49:52.935006 15192 tablet_service.cc:1467] Processing CreateTablet for tablet 493096d0edae42e1958204676bc8cb06 (DEFAULT_TABLE table=test-table [id=c6a8e0663f7343cea3aea3fb83a54da4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:49:52.936360 15192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493096d0edae42e1958204676bc8cb06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:49:52.944713 15273 tablet_bootstrap.cc:492] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d: Bootstrap starting.
I20260929 20:49:52.948484 15273 tablet_bootstrap.cc:654] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d: Neither blocks nor log segments found. Creating new log.
I20260929 20:49:52.949601 15273 log.cc:824] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:49:52.949775 15274 tablet_bootstrap.cc:492] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c: Bootstrap starting.
I20260929 20:49:52.952064 15276 tablet_bootstrap.cc:492] T 493096d0edae42e1958204676bc8cb06 P b45ac457da574e91a068aa2bb2127d5a: Bootstrap starting.
I20260929 20:49:52.953040 15274 tablet_bootstrap.cc:654] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c: Neither blocks nor log segments found. Creating new log.
I20260929 20:49:52.953169 15273 tablet_bootstrap.cc:492] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d: No bootstrap required, opened a new log
I20260929 20:49:52.953425 15273 ts_tablet_manager.cc:1397] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 20:49:52.954195 15274 log.cc:824] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:49:52.954600 15276 tablet_bootstrap.cc:654] T 493096d0edae42e1958204676bc8cb06 P b45ac457da574e91a068aa2bb2127d5a: Neither blocks nor log segments found. Creating new log.
I20260929 20:49:52.955722 15276 log.cc:824] T 493096d0edae42e1958204676bc8cb06 P b45ac457da574e91a068aa2bb2127d5a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:49:52.958282 15274 tablet_bootstrap.cc:492] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c: No bootstrap required, opened a new log
I20260929 20:49:52.958503 15274 ts_tablet_manager.cc:1397] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260929 20:49:52.958568 15276 tablet_bootstrap.cc:492] T 493096d0edae42e1958204676bc8cb06 P b45ac457da574e91a068aa2bb2127d5a: No bootstrap required, opened a new log
I20260929 20:49:52.958845 15276 ts_tablet_manager.cc:1397] T 493096d0edae42e1958204676bc8cb06 P b45ac457da574e91a068aa2bb2127d5a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260929 20:49:52.963948 15273 raft_consensus.cc:348] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 } } peers { permanent_uuid: "5583b1ffe1694d959621cb3540c7057d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37283 } } peers { permanent_uuid: "b45ac457da574e91a068aa2bb2127d5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33365 } }
I20260929 20:49:52.964383 15273 raft_consensus.cc:374] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:49:52.964512 15273 raft_consensus.cc:729] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5583b1ffe1694d959621cb3540c7057d, State: Initialized, Role: FOLLOWER
I20260929 20:49:52.965131 15273 consensus_queue.cc:260] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 } } peers { permanent_uuid: "5583b1ffe1694d959621cb3540c7057d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37283 } } peers { permanent_uuid: "b45ac457da574e91a068aa2bb2127d5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33365 } }
I20260929 20:49:52.965618 15274 raft_consensus.cc:348] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 } } peers { permanent_uuid: "5583b1ffe1694d959621cb3540c7057d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37283 } } peers { permanent_uuid: "b45ac457da574e91a068aa2bb2127d5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33365 } }
I20260929 20:49:52.966030 15274 raft_consensus.cc:374] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:49:52.966154 15274 raft_consensus.cc:729] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65fab7ae9a23408a994ea10f870f1d6c, State: Initialized, Role: FOLLOWER
I20260929 20:49:52.966710 15274 consensus_queue.cc:260] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 } } peers { permanent_uuid: "5583b1ffe1694d959621cb3540c7057d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37283 } } peers { permanent_uuid: "b45ac457da574e91a068aa2bb2127d5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33365 } }
I20260929 20:49:52.967762 15126 heartbeater.cc:503] Master 127.0.97.62:44053 was elected leader, sending a full tablet report...
I20260929 20:49:52.968180 15274 ts_tablet_manager.cc:1428] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260929 20:49:52.970567 15276 raft_consensus.cc:348] T 493096d0edae42e1958204676bc8cb06 P b45ac457da574e91a068aa2bb2127d5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 } } peers { permanent_uuid: "5583b1ffe1694d959621cb3540c7057d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37283 } } peers { permanent_uuid: "b45ac457da574e91a068aa2bb2127d5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33365 } }
I20260929 20:49:52.971078 15276 raft_consensus.cc:374] T 493096d0edae42e1958204676bc8cb06 P b45ac457da574e91a068aa2bb2127d5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:49:52.971240 15276 raft_consensus.cc:729] T 493096d0edae42e1958204676bc8cb06 P b45ac457da574e91a068aa2bb2127d5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45ac457da574e91a068aa2bb2127d5a, State: Initialized, Role: FOLLOWER
I20260929 20:49:52.971851 14994 heartbeater.cc:503] Master 127.0.97.62:44053 was elected leader, sending a full tablet report...
I20260929 20:49:52.972504 15273 ts_tablet_manager.cc:1428] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d: Time spent starting tablet: real 0.019s user 0.002s sys 0.017s
I20260929 20:49:52.972677 15276 consensus_queue.cc:260] T 493096d0edae42e1958204676bc8cb06 P b45ac457da574e91a068aa2bb2127d5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 } } peers { permanent_uuid: "5583b1ffe1694d959621cb3540c7057d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37283 } } peers { permanent_uuid: "b45ac457da574e91a068aa2bb2127d5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33365 } }
I20260929 20:49:52.974354 15257 heartbeater.cc:503] Master 127.0.97.62:44053 was elected leader, sending a full tablet report...
I20260929 20:49:52.975000 15276 ts_tablet_manager.cc:1428] T 493096d0edae42e1958204676bc8cb06 P b45ac457da574e91a068aa2bb2127d5a: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
W20260929 20:49:52.989797 14995 tablet.cc:2369] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:49:52.992286 15279 raft_consensus.cc:482] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:49:52.992518 15279 raft_consensus.cc:504] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 } } peers { permanent_uuid: "5583b1ffe1694d959621cb3540c7057d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37283 } } peers { permanent_uuid: "b45ac457da574e91a068aa2bb2127d5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33365 } }
I20260929 20:49:52.995338 15279 leader_election.cc:288] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65fab7ae9a23408a994ea10f870f1d6c (127.0.97.2:35321), b45ac457da574e91a068aa2bb2127d5a (127.0.97.3:33365)
I20260929 20:49:52.997970 15280 raft_consensus.cc:482] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:49:52.998224 15280 raft_consensus.cc:504] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 } } peers { permanent_uuid: "5583b1ffe1694d959621cb3540c7057d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37283 } } peers { permanent_uuid: "b45ac457da574e91a068aa2bb2127d5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33365 } }
I20260929 20:49:52.999961 15280 leader_election.cc:288] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5583b1ffe1694d959621cb3540c7057d (127.0.97.1:37283), b45ac457da574e91a068aa2bb2127d5a (127.0.97.3:33365)
I20260929 20:49:53.001860 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:52.994909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49805 (local address 127.0.97.3:33365)
0929 20:49:52.995294 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:52.995300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:52.996167 (+ 867us) server_negotiation.cc:407] Connection header received
0929 20:49:52.996405 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:52.996409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:52.996464 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:52.996536 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:49:52.996993 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.997702 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.999616 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:53.000664 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:53.000787 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:53.000894 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:53.001412 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 20:49:53.001551 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269}
I20260929 20:49:53.002952 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:52.994467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33365 (local address 127.0.97.1:49805)
0929 20:49:52.996060 (+ 1593us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:52.996083 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:52.996270 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:52.996561 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:52.996566 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:52.996588 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:49:52.996863 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:52.996872 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.997841 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.997847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:52.999465 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:52.999477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:53.001033 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:53.001040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:53.001227 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:53.001261 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:49:53.002395 (+ 1134us) client_negotiation.cc:240] Negotiation successful
0929 20:49:53.002556 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1451,"mutex_wait_us":1,"thread_start_us":1543,"threads_started":1}
I20260929 20:49:53.003230 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:52.994779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34615 (local address 127.0.97.2:35321)
0929 20:49:52.995150 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:52.995157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:52.996408 (+ 1251us) server_negotiation.cc:407] Connection header received
0929 20:49:52.996640 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:52.996646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:52.996715 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:52.996772 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:49:52.997475 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.998428 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:53.000746 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:53.002011 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:53.002109 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:53.002204 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:53.002810 (+ 606us) server_negotiation.cc:299] Negotiation successful
0929 20:49:53.002945 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260929 20:49:53.004048 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:52.994469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35321 (local address 127.0.97.1:34615)
0929 20:49:52.996314 (+ 1845us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:52.996330 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:52.996498 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:52.996963 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:52.996966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:52.996987 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:49:52.997330 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:52.997338 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:52.998579 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:52.998584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:53.000586 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:53.000600 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:53.002426 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:53.002432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:53.002623 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:53.002664 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 20:49:53.003520 (+ 856us) client_negotiation.cc:240] Negotiation successful
0929 20:49:53.003671 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1727,"mutex_wait_us":838,"thread_start_us":390,"threads_started":1}
I20260929 20:49:53.004149 15212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493096d0edae42e1958204676bc8cb06" candidate_uuid: "5583b1ffe1694d959621cb3540c7057d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45ac457da574e91a068aa2bb2127d5a" is_pre_election: true
I20260929 20:49:53.004733 15212 raft_consensus.cc:2454] T 493096d0edae42e1958204676bc8cb06 P b45ac457da574e91a068aa2bb2127d5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5583b1ffe1694d959621cb3540c7057d in term 0.
I20260929 20:49:53.005625 14882 leader_election.cc:302] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5583b1ffe1694d959621cb3540c7057d, b45ac457da574e91a068aa2bb2127d5a; no voters:
I20260929 20:49:53.006019 15279 raft_consensus.cc:2781] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:49:53.006142 15279 raft_consensus.cc:482] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:49:53.006250 15279 raft_consensus.cc:3037] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:49:53.006081 15081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493096d0edae42e1958204676bc8cb06" candidate_uuid: "5583b1ffe1694d959621cb3540c7057d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65fab7ae9a23408a994ea10f870f1d6c" is_pre_election: true
I20260929 20:49:53.006584 15081 raft_consensus.cc:2454] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5583b1ffe1694d959621cb3540c7057d in term 0.
I20260929 20:49:53.009445 15279 raft_consensus.cc:504] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 } } peers { permanent_uuid: "5583b1ffe1694d959621cb3540c7057d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37283 } } peers { permanent_uuid: "b45ac457da574e91a068aa2bb2127d5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33365 } }
I20260929 20:49:53.010115 15279 leader_election.cc:288] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d [CANDIDATE]: Term 1 election: Requested vote from peers 65fab7ae9a23408a994ea10f870f1d6c (127.0.97.2:35321), b45ac457da574e91a068aa2bb2127d5a (127.0.97.3:33365)
I20260929 20:49:53.010715 15212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493096d0edae42e1958204676bc8cb06" candidate_uuid: "5583b1ffe1694d959621cb3540c7057d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45ac457da574e91a068aa2bb2127d5a"
I20260929 20:49:53.010757 15081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493096d0edae42e1958204676bc8cb06" candidate_uuid: "5583b1ffe1694d959621cb3540c7057d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65fab7ae9a23408a994ea10f870f1d6c"
I20260929 20:49:53.010934 15081 raft_consensus.cc:3037] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:49:53.010927 15212 raft_consensus.cc:3037] T 493096d0edae42e1958204676bc8cb06 P b45ac457da574e91a068aa2bb2127d5a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:49:53.011770 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:53.000985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47979 (local address 127.0.97.3:33365)
0929 20:49:53.002072 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:53.002077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:53.005607 (+ 3530us) server_negotiation.cc:407] Connection header received
0929 20:49:53.005878 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:53.005883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:53.005941 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:53.005992 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:49:53.006791 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:53.007838 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:53.009662 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:53.010572 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:53.010689 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:53.010806 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:53.011365 (+ 559us) server_negotiation.cc:299] Negotiation successful
0929 20:49:53.011501 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":990,"thread_start_us":3570,"threads_started":1}
I20260929 20:49:53.012485 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:53.000574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47299 (local address 127.0.97.1:37283)
0929 20:49:53.000785 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:53.000791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:53.005318 (+ 4527us) server_negotiation.cc:407] Connection header received
0929 20:49:53.005654 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:53.005658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:53.005719 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:53.005780 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:49:53.006638 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:53.007772 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:53.010275 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:53.011426 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:53.011525 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:53.011603 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:53.012158 (+ 555us) server_negotiation.cc:299] Negotiation successful
0929 20:49:53.012277 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 20:49:53.012512 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:53.000347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33365 (local address 127.0.97.2:47979)
0929 20:49:53.005524 (+ 5177us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:53.005543 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:53.005685 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:53.006148 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:53.006152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:53.006172 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:49:53.006593 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:53.006603 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:53.007956 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:53.007960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:53.009525 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:53.009536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:53.010962 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:53.010968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:53.011177 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:53.011215 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 20:49:53.011966 (+ 751us) client_negotiation.cc:240] Negotiation successful
0929 20:49:53.012101 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5036,"mutex_wait_us":23,"thread_start_us":4559,"threads_started":1}
I20260929 20:49:53.013183 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:53.000150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37283 (local address 127.0.97.2:47299)
0929 20:49:53.005215 (+ 5065us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:53.005232 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:49:53.005484 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:49:53.005942 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:49:53.005946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:49:53.005987 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:49:53.006473 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:53.006485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:53.007924 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:53.007930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:53.010084 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:49:53.010099 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:49:53.011787 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:49:53.011793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:49:53.011972 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:53.012006 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:49:53.012618 (+ 612us) client_negotiation.cc:240] Negotiation successful
0929 20:49:53.012752 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4937,"mutex_wait_us":57,"thread_start_us":5044,"threads_started":1}
I20260929 20:49:53.013163 15211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493096d0edae42e1958204676bc8cb06" candidate_uuid: "65fab7ae9a23408a994ea10f870f1d6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b45ac457da574e91a068aa2bb2127d5a" is_pre_election: true
I20260929 20:49:53.013970 14949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493096d0edae42e1958204676bc8cb06" candidate_uuid: "65fab7ae9a23408a994ea10f870f1d6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5583b1ffe1694d959621cb3540c7057d" is_pre_election: true
I20260929 20:49:53.014102 15212 raft_consensus.cc:2454] T 493096d0edae42e1958204676bc8cb06 P b45ac457da574e91a068aa2bb2127d5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5583b1ffe1694d959621cb3540c7057d in term 1.
I20260929 20:49:53.014364 14949 raft_consensus.cc:2379] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 65fab7ae9a23408a994ea10f870f1d6c in current term 1: Already voted for candidate 5583b1ffe1694d959621cb3540c7057d in this term.
I20260929 20:49:53.014675 15081 raft_consensus.cc:2454] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5583b1ffe1694d959621cb3540c7057d in term 1.
I20260929 20:49:53.015082 14885 leader_election.cc:302] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5583b1ffe1694d959621cb3540c7057d, 65fab7ae9a23408a994ea10f870f1d6c; no voters:
I20260929 20:49:53.015419 15279 raft_consensus.cc:2781] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:49:53.015602 15016 leader_election.cc:302] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 65fab7ae9a23408a994ea10f870f1d6c; no voters: 5583b1ffe1694d959621cb3540c7057d, b45ac457da574e91a068aa2bb2127d5a
I20260929 20:49:53.015901 15279 raft_consensus.cc:686] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d [term 1 LEADER]: Becoming Leader. State: Replica: 5583b1ffe1694d959621cb3540c7057d, State: Running, Role: LEADER
I20260929 20:49:53.015982 15280 raft_consensus.cc:2726] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 20:49:53.016398 15279 consensus_queue.cc:237] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 } } peers { permanent_uuid: "5583b1ffe1694d959621cb3540c7057d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37283 } } peers { permanent_uuid: "b45ac457da574e91a068aa2bb2127d5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33365 } }
I20260929 20:49:53.020439 14809 catalog_manager.cc:5495] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5583b1ffe1694d959621cb3540c7057d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "5583b1ffe1694d959621cb3540c7057d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5583b1ffe1694d959621cb3540c7057d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b45ac457da574e91a068aa2bb2127d5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33365 } health_report { overall_health: UNKNOWN } } }
I20260929 20:49:53.050390 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:53.045589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37144 (local address 127.0.97.1:37283)
0929 20:49:53.045754 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:53.045758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:53.045776 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:49:53.045862 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:53.045865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:53.045940 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:53.046058 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:49:53.046396 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:53.047160 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:53.048595 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:53.048977 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:53.049065 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:53.049468 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:49:53.049944 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:49:53.050054 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:53.050089 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 20:49:53.050172 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 20:49:53.058282 15212 raft_consensus.cc:1261] T 493096d0edae42e1958204676bc8cb06 P b45ac457da574e91a068aa2bb2127d5a [term 1 FOLLOWER]: Refusing update from remote peer 5583b1ffe1694d959621cb3540c7057d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:49:53.058347 15081 raft_consensus.cc:1261] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c [term 1 FOLLOWER]: Refusing update from remote peer 5583b1ffe1694d959621cb3540c7057d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:49:53.059139 15287 consensus_queue.cc:1035] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b45ac457da574e91a068aa2bb2127d5a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:49:53.059440 15279 consensus_queue.cc:1035] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d [LEADER]: Connected to new peer: Peer: permanent_uuid: "65fab7ae9a23408a994ea10f870f1d6c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:49:53.069725 15290 mvcc.cc:205] Tried to move back new op lower bound from 7334768611557941248 to 7334768611398569984. Current Snapshot: MvccSnapshot[applied={T|T < 7334768611557941248}]
I20260929 20:49:53.083634 14809 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:49:53.083859 14809 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:49:53.086907 14809 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8: Sending DeleteTablet for 3 replicas of tablet 493096d0edae42e1958204676bc8cb06
I20260929 20:49:53.088032 15061 tablet_service.cc:1514] Processing DeleteTablet for tablet 493096d0edae42e1958204676bc8cb06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:49:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59004
I20260929 20:49:53.088083 15192 tablet_service.cc:1514] Processing DeleteTablet for tablet 493096d0edae42e1958204676bc8cb06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:49:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40110
I20260929 20:49:53.088907 15299 tablet_replica.cc:330] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c: stopping tablet replica
I20260929 20:49:53.089160 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 14846
I20260929 20:49:53.089066 14929 tablet_service.cc:1514] Processing DeleteTablet for tablet 493096d0edae42e1958204676bc8cb06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:49:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37132
I20260929 20:49:53.089197 15300 tablet_replica.cc:330] T 493096d0edae42e1958204676bc8cb06 P b45ac457da574e91a068aa2bb2127d5a: stopping tablet replica
I20260929 20:49:53.089581 15300 raft_consensus.cc:2229] T 493096d0edae42e1958204676bc8cb06 P b45ac457da574e91a068aa2bb2127d5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:49:53.090091 15300 raft_consensus.cc:2258] T 493096d0edae42e1958204676bc8cb06 P b45ac457da574e91a068aa2bb2127d5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:49:53.090152 15301 tablet_replica.cc:330] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d: stopping tablet replica
I20260929 20:49:53.090618 15301 raft_consensus.cc:2229] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:49:53.091065 15299 raft_consensus.cc:2229] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:49:53.091324 15301 raft_consensus.cc:2258] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:49:53.091552 15299 raft_consensus.cc:2258] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:49:53.092542 15300 ts_tablet_manager.cc:1905] T 493096d0edae42e1958204676bc8cb06 P b45ac457da574e91a068aa2bb2127d5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:49:53.093295 15301 ts_tablet_manager.cc:1905] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:49:53.094695 15299 ts_tablet_manager.cc:1905] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:49:53.099318 15301 ts_tablet_manager.cc:1918] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:49:53.099466 15301 log.cc:1196] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/493096d0edae42e1958204676bc8cb06
I20260929 20:49:53.099946 15301 ts_tablet_manager.cc:1939] T 493096d0edae42e1958204676bc8cb06 P 5583b1ffe1694d959621cb3540c7057d: Deleting consensus metadata
I20260929 20:49:53.101707 14795 catalog_manager.cc:4841] TS 5583b1ffe1694d959621cb3540c7057d (127.0.97.1:37283): tablet 493096d0edae42e1958204676bc8cb06 (table test-table [id=c6a8e0663f7343cea3aea3fb83a54da4]) successfully deleted
I20260929 20:49:53.103138 15300 ts_tablet_manager.cc:1918] T 493096d0edae42e1958204676bc8cb06 P b45ac457da574e91a068aa2bb2127d5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:49:53.103865 15300 log.cc:1196] T 493096d0edae42e1958204676bc8cb06 P b45ac457da574e91a068aa2bb2127d5a: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/493096d0edae42e1958204676bc8cb06
I20260929 20:49:53.104249 15299 ts_tablet_manager.cc:1918] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:49:53.104504 15299 log.cc:1196] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/493096d0edae42e1958204676bc8cb06
I20260929 20:49:53.104789 15300 ts_tablet_manager.cc:1939] T 493096d0edae42e1958204676bc8cb06 P b45ac457da574e91a068aa2bb2127d5a: Deleting consensus metadata
I20260929 20:49:53.105130 15299 ts_tablet_manager.cc:1939] T 493096d0edae42e1958204676bc8cb06 P 65fab7ae9a23408a994ea10f870f1d6c: Deleting consensus metadata
I20260929 20:49:53.105942 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:53.090207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37152 (local address 127.0.97.1:37283)
0929 20:49:53.090358 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:53.090363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:53.090382 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:49:53.090480 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:53.090485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:53.090602 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:53.090715 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:53.092493 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:53.093454 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:53.095314 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:53.095928 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:53.096052 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:53.099253 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:53.099270 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:53.099288 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:53.099333 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:53.103140 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:53.103741 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:53.103749 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:53.103753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:53.103929 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:53.104311 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:53.104316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:53.104319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:53.104833 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:53.105097 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:53.105402 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 20:49:53.105627 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:49:53.106825 14989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:49:53.107556 14793 catalog_manager.cc:4841] TS b45ac457da574e91a068aa2bb2127d5a (127.0.97.3:33365): tablet 493096d0edae42e1958204676bc8cb06 (table test-table [id=c6a8e0663f7343cea3aea3fb83a54da4]) successfully deleted
I20260929 20:49:53.107697 14796 catalog_manager.cc:4841] TS 65fab7ae9a23408a994ea10f870f1d6c (127.0.97.2:35321): tablet 493096d0edae42e1958204676bc8cb06 (table test-table [id=c6a8e0663f7343cea3aea3fb83a54da4]) successfully deleted
I20260929 20:49:53.201758 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 14846
I20260929 20:49:53.216343 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 14998
I20260929 20:49:53.232278 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:53.217124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59012 (local address 127.0.97.2:35321)
0929 20:49:53.217283 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:53.217287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:53.217306 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:49:53.217411 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:53.217415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:53.217485 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:53.217567 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:53.218808 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:53.219645 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:53.220814 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:53.221216 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:53.221308 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:53.227488 (+ 6180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:53.227501 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:53.227517 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:53.227548 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:53.229970 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:53.230526 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:53.230532 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:53.230534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:53.230658 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:53.230926 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:53.230928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:53.230930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:53.231397 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:53.231613 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:53.231823 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 20:49:53.232001 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:49:53.232897 15121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:49:53.321907 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 14998
I20260929 20:49:53.335609 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 15129
I20260929 20:49:53.347644 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:53.336271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40112 (local address 127.0.97.3:33365)
0929 20:49:53.336421 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 20:49:53.336425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:49:53.336485 (+ 60us) server_negotiation.cc:407] Connection header received
0929 20:49:53.336659 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:49:53.336664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:49:53.336743 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:49:53.336820 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:49:53.338070 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:53.338758 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:53.340048 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:49:53.340462 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:49:53.340554 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:49:53.342776 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:49:53.342792 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:49:53.342809 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:49:53.342839 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:49:53.345428 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:53.345997 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:53.346003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:53.346005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:53.346120 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:49:53.346393 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:49:53.346396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:49:53.346398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:49:53.346763 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:49:53.346973 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 20:49:53.347234 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 20:49:53.347394 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 20:49:53.348290 15252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:49:53.436373 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 15129
I20260929 20:49:53.449751 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 14776
I20260929 20:49:53.450879 14839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:49:53.559911 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 14776
I20260929 20:49:53.576921 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44053
--webserver_interface=127.0.97.62
--webserver_port=33225
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:49:53.886859 15306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:53.887187 15306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:53.887292 15306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:53.894201 15306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:49:53.894286 15306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:53.894336 15306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:53.894388 15306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:49:53.894429 15306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:49:53.904111 15306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:53.905355 15306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:53.906589 15306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:53.920516 15311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:53.923296 15312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:53.924860 15315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:53.925705 15306 server_base.cc:1034] running on GCE node
I20260929 20:49:53.926736 15306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:53.928596 15306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:53.929850 15306 hybrid_clock.cc:648] HybridClock initialized: now 1790714993929792 us; error 42 us; skew 500 ppm
I20260929 20:49:53.930305 15306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:53 dist-test-slave-blv7 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:53.934322 15306 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:49:53.936182 15306 webserver.cc:460] Webserver started at http://127.0.97.62:33225/ using document root <none> and password file <none>
I20260929 20:49:53.937050 15306 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:53.937152 15306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:49:53.944413 15306 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:49:53.947970 15321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:53.949616 15306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:49:53.949790 15306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "6c1bff5661b649058bc06d5ff580cbd8"
format_stamp: "Formatted at 2026-09-29 20:49:48 on dist-test-slave-blv7"
server_key: "2dd1f5b6c21581dfd5c5536e4b521788"
server_key_iv: "95b7a7e178569a52962e6f04004a3495"
server_key_version: "encryptionkey@0"
I20260929 20:49:53.951167 15306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:53.965065 15306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:53.968791 15306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:53.969008 15306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:53.985801 15306 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44053
I20260929 20:49:53.985821 15372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44053 every 8 connection(s)
I20260929 20:49:53.987356 15306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:49:53.993958 15373 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:49:53.995235 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 15306
I20260929 20:49:53.996196 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:37283
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:44053
--webserver_port=42609
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:49:53.998131 15373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8: Bootstrap starting.
I20260929 20:49:54.018941 15373 log.cc:824] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8: Log is configured to *not* fsync() on all Append() calls
I20260929 20:49:54.034569 15373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:49:54.035105 15373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8: Bootstrap complete.
I20260929 20:49:54.043835 15373 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1bff5661b649058bc06d5ff580cbd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44053 } }
I20260929 20:49:54.044726 15373 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c1bff5661b649058bc06d5ff580cbd8, State: Initialized, Role: FOLLOWER
I20260929 20:49:54.045326 15373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1bff5661b649058bc06d5ff580cbd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44053 } }
I20260929 20:49:54.045539 15373 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:49:54.045650 15373 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:49:54.045804 15373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 2 FOLLOWER]: Advancing to term 3
I20260929 20:49:54.049422 15373 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1bff5661b649058bc06d5ff580cbd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44053 } }
I20260929 20:49:54.049866 15373 leader_election.cc:302] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c1bff5661b649058bc06d5ff580cbd8; no voters:
I20260929 20:49:54.050469 15373 leader_election.cc:288] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 20:49:54.050594 15377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 20:49:54.051652 15377 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [term 3 LEADER]: Becoming Leader. State: Replica: 6c1bff5661b649058bc06d5ff580cbd8, State: Running, Role: LEADER
I20260929 20:49:54.052165 15377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1bff5661b649058bc06d5ff580cbd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44053 } }
I20260929 20:49:54.052521 15373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:49:54.054533 15378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6c1bff5661b649058bc06d5ff580cbd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1bff5661b649058bc06d5ff580cbd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44053 } } }
I20260929 20:49:54.054893 15378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [sys.catalog]: This master's current role is: LEADER
I20260929 20:49:54.058689 15379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c1bff5661b649058bc06d5ff580cbd8. Latest consensus state: current_term: 3 leader_uuid: "6c1bff5661b649058bc06d5ff580cbd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1bff5661b649058bc06d5ff580cbd8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44053 } } }
I20260929 20:49:54.058971 15379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c1bff5661b649058bc06d5ff580cbd8 [sys.catalog]: This master's current role is: LEADER
I20260929 20:49:54.060014 15382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:49:54.068691 15382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:49:54.070175 15382 catalog_manager.cc:1259] Loaded cluster ID: a78f75b8fcdc43fa8b5bc988852ae480
I20260929 20:49:54.070338 15382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 20:49:54.072381 15382 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa2a0bab9ad google::LogMessage::Fail() at ??:0
@ 0x7fa2a0bae88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa2a0bab4de google::LogMessage::Flush() at ??:0
@ 0x7fa2a0baca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa2b65de512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa2b65db976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa2b6663610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa2b6663439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa2a37f7345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa2a37fa91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa2a37fa799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa2a37d5eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa2a6aaf6db start_thread at ??:0
@ 0x7fa29fa4561f clone at ??:0
W20260929 20:49:54.372952 15375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:54.373217 15375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:54.373311 15375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:54.381377 15375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:54.381482 15375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:54.381603 15375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:49:54.391295 15375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:37283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42609
--tserver_master_addrs=127.0.97.62:44053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:54.392520 15375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:54.393775 15375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:54.406213 15414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:54.406265 15415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:54.406294 15417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:54.407333 15375 server_base.cc:1034] running on GCE node
I20260929 20:49:54.407878 15375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:54.408756 15375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:54.409950 15375 hybrid_clock.cc:648] HybridClock initialized: now 1790714994409893 us; error 48 us; skew 500 ppm
I20260929 20:49:54.410241 15375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:54 dist-test-slave-blv7 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:54.413100 15375 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:49:54.414378 15375 webserver.cc:460] Webserver started at http://127.0.97.1:42609/ using document root <none> and password file <none>
I20260929 20:49:54.415006 15375 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:54.415163 15375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:49:54.420102 15375 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 20:49:54.422573 15424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:54.423825 15375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:49:54.423964 15375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "5583b1ffe1694d959621cb3540c7057d"
format_stamp: "Formatted at 2026-09-29 20:49:49 on dist-test-slave-blv7"
server_key: "6c0daeb80ea2f4122524fcacb6de0052"
server_key_iv: "7d16113fe13386f1bab159f5400c0ffa"
server_key_version: "encryptionkey@0"
I20260929 20:49:54.424888 15375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:54.450495 15375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:54.454469 15375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:54.454727 15375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:54.455760 15375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:49:54.457186 15375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:49:54.457284 15375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:54.457394 15375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:49:54.457466 15375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:54.487259 15375 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37283
I20260929 20:49:54.487317 15536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37283 every 8 connection(s)
I20260929 20:49:54.488682 15375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:49:54.489228 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 15375
I20260929 20:49:54.489950 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:35321
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:44053
--webserver_port=38367
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:49:54.493978 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:54.491988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.1:40153)
0929 20:49:54.492754 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:54.492965 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":295,"threads_started":1}
W20260929 20:49:54.494635 15537 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44053 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:44053: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
I20260929 20:49:54.495584 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:54.495225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.1:34731)
0929 20:49:54.495376 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:54.495451 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 20:49:54.496433 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:54.496147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.1:51197)
0929 20:49:54.496254 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:54.496327 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 20:49:54.496667 15537 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:49:54.809911 15540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:54.810230 15540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:54.810357 15540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:54.817752 15540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:54.817868 15540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:54.818030 15540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:49:54.827881 15540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:35321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=38367
--tserver_master_addrs=127.0.97.62:44053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:54.829279 15540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:54.830529 15540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:54.843094 15546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:54.843030 15548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:54.843070 15545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:54.848376 15540 server_base.cc:1034] running on GCE node
I20260929 20:49:54.848959 15540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:54.849887 15540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:54.851121 15540 hybrid_clock.cc:648] HybridClock initialized: now 1790714994851049 us; error 61 us; skew 500 ppm
I20260929 20:49:54.851416 15540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:54 dist-test-slave-blv7 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:54.854539 15540 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:49:54.856093 15540 webserver.cc:460] Webserver started at http://127.0.97.2:38367/ using document root <none> and password file <none>
I20260929 20:49:54.856787 15540 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:54.856902 15540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:49:54.862241 15540 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 20:49:54.865072 15555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:54.866501 15540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:49:54.866683 15540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "65fab7ae9a23408a994ea10f870f1d6c"
format_stamp: "Formatted at 2026-09-29 20:49:49 on dist-test-slave-blv7"
server_key: "1424bcd628ad6b37c2039c5ef19a7578"
server_key_iv: "53ac5d2d6462b72ea1689dffc96b3cd2"
server_key_version: "encryptionkey@0"
I20260929 20:49:54.867709 15540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:54.879675 15540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:54.883543 15540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:54.883790 15540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:54.884725 15540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:49:54.885977 15540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:49:54.886076 15540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:54.886180 15540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:49:54.886248 15540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:54.916236 15540 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35321
I20260929 20:49:54.916299 15667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35321 every 8 connection(s)
I20260929 20:49:54.917658 15540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:49:54.922159 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 15540
I20260929 20:49:54.922712 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:54.920561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.2:43309)
0929 20:49:54.921276 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:54.921476 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":267,"threads_started":1}
I20260929 20:49:54.922896 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:33365
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:44053
--webserver_port=40453
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:49:54.923470 15668 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44053 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:44053: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
I20260929 20:49:54.924396 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:54.923977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.2:41819)
0929 20:49:54.924181 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:54.924265 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 20:49:54.925276 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:54.924932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.2:48931)
0929 20:49:54.925062 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:54.925147 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 20:49:54.925520 15668 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:49:55.229247 15671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:49:55.229521 15671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:49:55.229660 15671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:49:55.236766 15671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:49:55.236899 15671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:49:55.237056 15671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:49:55.246832 15671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:33365
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=40453
--tserver_master_addrs=127.0.97.62:44053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:49:55.248088 15671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:49:55.249342 15671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:49:55.261966 15677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:55.262036 15676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:49:55.262009 15679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:49:55.262801 15671 server_base.cc:1034] running on GCE node
I20260929 20:49:55.263463 15671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:49:55.265301 15671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:49:55.266542 15671 hybrid_clock.cc:648] HybridClock initialized: now 1790714995266481 us; error 54 us; skew 500 ppm
I20260929 20:49:55.266888 15671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:49:55 dist-test-slave-blv7 krb5kdc[14226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790714995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:49:55.270020 15671 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:49:55.271552 15671 webserver.cc:460] Webserver started at http://127.0.97.3:40453/ using document root <none> and password file <none>
I20260929 20:49:55.272255 15671 fs_manager.cc:359] Metadata directory not provided
I20260929 20:49:55.272378 15671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:49:55.277508 15671 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 20:49:55.280061 15686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:55.281311 15671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:49:55.281452 15671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "b45ac457da574e91a068aa2bb2127d5a"
format_stamp: "Formatted at 2026-09-29 20:49:50 on dist-test-slave-blv7"
server_key: "98e30136355852f3f8b6fa8fecb807a3"
server_key_iv: "c0e719525c87e7633023209bab206075"
server_key_version: "encryptionkey@0"
I20260929 20:49:55.282388 15671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:49:55.300420 15671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:49:55.304447 15671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:49:55.304695 15671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:49:55.305613 15671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:49:55.306877 15671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:49:55.306980 15671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:55.307120 15671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:49:55.307190 15671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:49:55.335490 15671 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33365
I20260929 20:49:55.335538 15798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33365 every 8 connection(s)
I20260929 20:49:55.336869 15671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:49:55.341590 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:55.339797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.3:48135)
0929 20:49:55.340516 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:55.340708 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":289,"threads_started":1}
W20260929 20:49:55.342307 15799 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44053 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:44053: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
I20260929 20:49:55.343128 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 15671
I20260929 20:49:55.343245 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:55.342855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.3:60359)
0929 20:49:55.343058 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:55.343137 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260929 20:49:55.344038 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:55.343742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.3:36543)
0929 20:49:55.343863 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:55.343934 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 20:49:55.344277 15799 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 20:49:55.497854 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:55.497145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.1:43033)
0929 20:49:55.497554 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:55.497634 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":200,"threads_started":1}
I20260929 20:49:55.926890 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:55.926103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.2:53657)
0929 20:49:55.926539 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:55.926669 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":204,"threads_started":1}
I20260929 20:49:56.345742 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:56.344927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.3:35299)
0929 20:49:56.345401 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:56.345493 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":238,"threads_started":1}
I20260929 20:49:56.499374 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:56.498665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.1:59991)
0929 20:49:56.499121 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:56.499203 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":202,"threads_started":1}
I20260929 20:49:56.928442 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:56.927787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.2:51849)
0929 20:49:56.928205 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:56.928280 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":195,"threads_started":1}
I20260929 20:49:57.347273 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:57.346565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.3:58465)
0929 20:49:57.346997 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:57.347111 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":220,"threads_started":1}
I20260929 20:49:57.500774 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:57.500129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.1:54417)
0929 20:49:57.500526 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:57.500599 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":209,"threads_started":1}
I20260929 20:49:57.930044 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:57.929363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.2:51545)
0929 20:49:57.929794 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:57.929870 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":228,"threads_started":1}
I20260929 20:49:58.348800 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:58.348125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.3:53777)
0929 20:49:58.348537 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:58.348622 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":213,"threads_started":1}
I20260929 20:49:58.502362 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:58.501611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.1:42601)
0929 20:49:58.502071 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:58.502165 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":230,"threads_started":1}
I20260929 20:49:58.931701 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:58.930927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.2:46819)
0929 20:49:58.931401 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:58.931487 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":247,"threads_started":1}
I20260929 20:49:59.350440 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:59.349706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.3:40989)
0929 20:49:59.350163 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:59.350248 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":228,"threads_started":1}
I20260929 20:49:59.504015 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:59.503315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.1:39137)
0929 20:49:59.503769 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:59.503850 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":223,"threads_started":1}
I20260929 20:49:59.933305 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:49:59.932604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.2:33743)
0929 20:49:59.933050 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 20:49:59.933132 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":232,"threads_started":1}
I20260929 20:50:00.352059 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:00.351368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.3:35897)
0929 20:50:00.351811 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:00.351894 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":222,"threads_started":1}
I20260929 20:50:00.505589 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:00.504899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.1:55205)
0929 20:50:00.505322 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:00.505434 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":239,"threads_started":1}
I20260929 20:50:00.935081 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:00.934266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.2:33587)
0929 20:50:00.934763 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:00.934859 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":240,"threads_started":1}
I20260929 20:50:01.353579 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:01.352872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.3:50273)
0929 20:50:01.353332 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:01.353421 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":253,"threads_started":1}
I20260929 20:50:01.507423 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:01.506575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.1:48585)
0929 20:50:01.507108 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:01.507214 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":215,"threads_started":1}
I20260929 20:50:01.936748 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:01.935925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.2:46325)
0929 20:50:01.936459 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:01.936539 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":232,"threads_started":1}
I20260929 20:50:02.355274 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:02.354448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.3:40553)
0929 20:50:02.354946 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:02.355050 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":256,"threads_started":1}
I20260929 20:50:02.509024 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:02.508323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.1:51063)
0929 20:50:02.508773 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:02.508858 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":220,"threads_started":1}
I20260929 20:50:02.938364 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:02.937589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.2:57141)
0929 20:50:02.938098 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:02.938186 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":262,"threads_started":1}
I20260929 20:50:03.356952 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:03.356193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.3:54075)
0929 20:50:03.356691 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:03.356785 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":266,"threads_started":1}
I20260929 20:50:03.510702 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:03.509910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.1:35461)
0929 20:50:03.510448 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:03.510532 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":279,"threads_started":1}
I20260929 20:50:03.940158 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:03.939445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.2:44621)
0929 20:50:03.939900 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:03.939985 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":222,"threads_started":1}
I20260929 20:50:04.358497 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:04.357790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.3:41093)
0929 20:50:04.358247 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:04.358327 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":209,"threads_started":1}
I20260929 20:50:04.512235 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:04.511556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.1:49897)
0929 20:50:04.511985 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:04.512067 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":224,"threads_started":1}
I20260929 20:50:04.941833 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:04.941077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.2:44815)
0929 20:50:04.941570 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:04.941654 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":284,"threads_started":1}
I20260929 20:50:05.360049 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:05.359383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.3:35211)
0929 20:50:05.359808 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:05.359890 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":232,"threads_started":1}
I20260929 20:50:05.513988 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:05.513198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.1:33465)
0929 20:50:05.513717 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:05.513812 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":236,"threads_started":1}
I20260929 20:50:05.943717 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:05.942898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.2:51855)
0929 20:50:05.943430 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:05.943514 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":296,"threads_started":1}
I20260929 20:50:06.361886 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:06.361095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.3:41659)
0929 20:50:06.361579 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:06.361693 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":241,"threads_started":1}
I20260929 20:50:06.515651 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:06.514930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.1:44555)
0929 20:50:06.515395 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:06.515491 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":236,"threads_started":1}
I20260929 20:50:06.945351 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:06.944615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.2:56221)
0929 20:50:06.945092 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:06.945173 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":252,"threads_started":1}
I20260929 20:50:07.363415 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:07.362725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.3:53521)
0929 20:50:07.363186 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:07.363263 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":222,"threads_started":1}
I20260929 20:50:07.517285 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:07.516545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.1:38061)
0929 20:50:07.517021 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:07.517103 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":240,"threads_started":1}
I20260929 20:50:07.946944 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:07.946201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.2:35071)
0929 20:50:07.946674 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:07.946758 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":216,"threads_started":1}
I20260929 20:50:08.364992 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:08.364283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.3:37437)
0929 20:50:08.364730 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:08.364810 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":222,"threads_started":1}
I20260929 20:50:08.518728 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:08.518051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.1:33905)
0929 20:50:08.518462 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:08.518540 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":201,"threads_started":1}
I20260929 20:50:08.948676 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:08.947863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.2:59115)
0929 20:50:08.948374 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:08.948465 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":259,"threads_started":1}
I20260929 20:50:09.366552 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:09.365849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.3:36777)
0929 20:50:09.366300 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:09.366379 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":233,"threads_started":1}
I20260929 20:50:09.520625 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:09.519840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.1:41687)
0929 20:50:09.520273 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:09.520390 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":216,"threads_started":1}
I20260929 20:50:09.950639 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:09.949858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.2:56055)
0929 20:50:09.950298 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:09.950399 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":222,"threads_started":1}
I20260929 20:50:10.348856 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 15375
I20260929 20:50:10.368399 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:10.367683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44053 (local address 127.0.97.3:58957)
0929 20:50:10.368117 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:10.368227 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:44053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":225,"threads_started":1}
I20260929 20:50:10.372212 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:10.349916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.0.97.1:37283)
0929 20:50:10.350473 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:10.350491 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:10.350539 (+ 48us) server_negotiation.cc:407] Connection header received
0929 20:50:10.352726 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:10.352736 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:10.353167 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:10.353474 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:50:10.354875 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:10.356280 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:10.357474 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:10.358142 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:10.358272 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:10.366311 (+ 8039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:50:10.366319 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:50:10.366337 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:50:10.366378 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:50:10.369340 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:10.369954 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:10.369961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:10.369968 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:10.370142 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:10.370417 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:10.370420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:10.370422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:10.370940 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:50:10.371183 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:10.371532 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 20:50:10.371941 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":239,"threads_started":1}
I20260929 20:50:10.373517 15532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:50:10.458277 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 15375
I20260929 20:50:10.472193 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 15540
I20260929 20:50:10.489624 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:10.473079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.0.97.2:35321)
0929 20:50:10.473645 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:10.473659 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:10.473696 (+ 37us) server_negotiation.cc:407] Connection header received
0929 20:50:10.475539 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:10.475549 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:10.476002 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:10.476309 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:50:10.477603 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:10.478851 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:10.480163 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:10.480841 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:10.480974 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:10.483514 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:50:10.483522 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:50:10.483537 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:50:10.483573 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:50:10.486372 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:10.487273 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:10.487279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:10.487286 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:10.487456 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:10.487837 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:10.487840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:10.487843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:10.488375 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:50:10.488603 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:10.488980 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 20:50:10.489348 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":249,"threads_started":1}
I20260929 20:50:10.490959 15663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:50:10.573643 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 15540
I20260929 20:50:10.587648 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 15671
I20260929 20:50:10.604611 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:10.588653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44272 (local address 127.0.97.3:33365)
0929 20:50:10.589317 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:10.589332 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:10.589373 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:50:10.590927 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:10.590937 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:10.591421 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:10.591706 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:50:10.592944 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:10.594137 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:10.595526 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:10.596200 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:10.596328 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:10.598978 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:50:10.598986 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:50:10.599001 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:50:10.599079 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:50:10.601714 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:10.602338 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:10.602344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:10.602350 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:10.602508 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:10.602832 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:10.602836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:10.602838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:10.603392 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:50:10.603630 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:10.603962 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 20:50:10.604316 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":327,"threads_started":1}
I20260929 20:50:10.605955 15794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:50:10.688473 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 15671
2026-09-29T20:50:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23707 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:50:10 dist-test-slave-blv7 krb5kdc[15856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:50:10 dist-test-slave-blv7 krb5kdc[15856](info): set up 2 sockets
Sep 29 20:50:10 dist-test-slave-blv7 krb5kdc[15856](info): commencing operation
krb5kdc: starting...
W20260929 20:50:11.691329 388 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:50:11 dist-test-slave-blv7 krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:50:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:50:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:50:11.906726 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42091
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:50:12.222021 15869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:50:12.222316 15869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:50:12.222446 15869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:50:12.229606 15869 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 20:50:12.229705 15869 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 20:50:12.229761 15869 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 20:50:12.229827 15869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:50:12.229897 15869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:50:12.229992 15869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:50:12.230050 15869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:50:12.230125 15869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:50:12.240085 15869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:50:12.241480 15869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:50:12.242873 15869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:50:12.253994 15877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:12.254069 15875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:12.254504 15874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:50:12.254748 15869 server_base.cc:1034] running on GCE node
I20260929 20:50:12.255532 15869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:50:12.256914 15869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:50:12.258117 15869 hybrid_clock.cc:648] HybridClock initialized: now 1790715012258059 us; error 43 us; skew 500 ppm
I20260929 20:50:12.258435 15869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:50:12 dist-test-slave-blv7 krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:50:12.261656 15869 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:50:12.263113 15869 webserver.cc:460] Webserver started at http://127.0.97.62:38015/ using document root <none> and password file <none>
I20260929 20:50:12.263799 15869 fs_manager.cc:359] Metadata directory not provided
I20260929 20:50:12.263886 15869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:50:12.264182 15869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:50:12.267071 15869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "cab36775e31448d180fd3daf0675b379"
format_stamp: "Formatted at 2026-09-29 20:50:12 on dist-test-slave-blv7"
server_key: "e8a34a1b6a61a012417edb04677f2da4"
server_key_iv: "0331bd94cabc8f407f23bba738328061"
server_key_version: "encryptionkey@0"
I20260929 20:50:12.267838 15869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "cab36775e31448d180fd3daf0675b379"
format_stamp: "Formatted at 2026-09-29 20:50:12 on dist-test-slave-blv7"
server_key: "e8a34a1b6a61a012417edb04677f2da4"
server_key_iv: "0331bd94cabc8f407f23bba738328061"
server_key_version: "encryptionkey@0"
I20260929 20:50:12.272784 15869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:50:12.276113 15884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:12.277374 15869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:50:12.277510 15869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "cab36775e31448d180fd3daf0675b379"
format_stamp: "Formatted at 2026-09-29 20:50:12 on dist-test-slave-blv7"
server_key: "e8a34a1b6a61a012417edb04677f2da4"
server_key_iv: "0331bd94cabc8f407f23bba738328061"
server_key_version: "encryptionkey@0"
I20260929 20:50:12.277700 15869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:50:12.294106 15869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:50:12.298380 15869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:50:12.298631 15869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:50:12.315546 15869 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42091
I20260929 20:50:12.315564 15935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42091 every 8 connection(s)
I20260929 20:50:12.316866 15869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:50:12.317303 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 15869
I20260929 20:50:12.317575 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:50:12.318290 388 external_mini_cluster.cc:1428] Setting key c2896031404b8a386b54f12e4d55078e
I20260929 20:50:12.321203 15936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:50:12 dist-test-slave-blv7 krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:50:12.331769 15936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379: Bootstrap starting.
I20260929 20:50:12.335107 15936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379: Neither blocks nor log segments found. Creating new log.
I20260929 20:50:12.336169 15936 log.cc:824] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379: Log is configured to *not* fsync() on all Append() calls
I20260929 20:50:12.339567 15936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379: No bootstrap required, opened a new log
I20260929 20:50:12.340241 15939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:12.320231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57162 (local address 127.0.97.62:42091)
0929 20:50:12.321015 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:12.321027 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:12.321065 (+ 38us) server_negotiation.cc:407] Connection header received
0929 20:50:12.322744 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:12.322766 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:12.323180 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:12.323544 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:50:12.324919 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:12.326240 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:12.327568 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:12.328218 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:12.328396 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:12.332500 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:50:12.332509 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:50:12.332524 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:50:12.332561 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:50:12.335724 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:12.336423 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:12.336432 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:12.336441 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:12.336668 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:12.337123 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:12.337127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:12.337129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:12.337768 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:50:12.338054 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:12.338455 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 20:50:12.338902 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":341,"threads_started":1}
I20260929 20:50:12.346902 15936 raft_consensus.cc:348] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab36775e31448d180fd3daf0675b379" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42091 } }
I20260929 20:50:12.347289 15936 raft_consensus.cc:374] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:50:12.347391 15936 raft_consensus.cc:729] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab36775e31448d180fd3daf0675b379, State: Initialized, Role: FOLLOWER
I20260929 20:50:12.347932 15936 consensus_queue.cc:260] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab36775e31448d180fd3daf0675b379" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42091 } }
I20260929 20:50:12.348133 15936 raft_consensus.cc:388] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:50:12.348259 15936 raft_consensus.cc:482] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:50:12.348403 15936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:50:12.350464 15936 raft_consensus.cc:504] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab36775e31448d180fd3daf0675b379" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42091 } }
I20260929 20:50:12.350927 15936 leader_election.cc:302] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cab36775e31448d180fd3daf0675b379; no voters:
I20260929 20:50:12.351511 15936 leader_election.cc:288] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:50:12.351630 15941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:50:12.352226 15941 raft_consensus.cc:686] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 1 LEADER]: Becoming Leader. State: Replica: cab36775e31448d180fd3daf0675b379, State: Running, Role: LEADER
I20260929 20:50:12.352814 15941 consensus_queue.cc:237] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab36775e31448d180fd3daf0675b379" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42091 } }
I20260929 20:50:12.353571 15936 sys_catalog.cc:564] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:50:12.355605 15943 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cab36775e31448d180fd3daf0675b379. Latest consensus state: current_term: 1 leader_uuid: "cab36775e31448d180fd3daf0675b379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab36775e31448d180fd3daf0675b379" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42091 } } }
I20260929 20:50:12.355979 15943 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [sys.catalog]: This master's current role is: LEADER
I20260929 20:50:12.356295 15942 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cab36775e31448d180fd3daf0675b379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab36775e31448d180fd3daf0675b379" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42091 } } }
I20260929 20:50:12.357147 15942 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [sys.catalog]: This master's current role is: LEADER
I20260929 20:50:12.357937 15949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:50:12.362273 15949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:50:12.370976 15949 catalog_manager.cc:1347] Generated new cluster ID: 2108cd2f26b5401bab680f8c3f0ae652
I20260929 20:50:12.371148 15949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:50:12.387948 15949 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:50:12.389331 15949 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:50:12.400184 15949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379: Generated new TSK 0
I20260929 20:50:12.400990 15949 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:50:12.466418 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42091
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:50:12.784936 15962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:50:12.785218 15962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:50:12.785349 15962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:50:12.792498 15962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:50:12.792614 15962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:50:12.792774 15962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:50:12.802436 15962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:50:12.803718 15962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:50:12.804967 15962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:50:12.817521 15967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:12.817480 15968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:12.817552 15970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:50:12.818219 15962 server_base.cc:1034] running on GCE node
I20260929 20:50:12.818784 15962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:50:12.819718 15962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:50:12.820914 15962 hybrid_clock.cc:648] HybridClock initialized: now 1790715012820861 us; error 45 us; skew 500 ppm
I20260929 20:50:12.821211 15962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:50:12 dist-test-slave-blv7 krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:50:12.824156 15962 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:50:12.825501 15962 webserver.cc:460] Webserver started at http://127.0.97.1:41535/ using document root <none> and password file <none>
I20260929 20:50:12.826148 15962 fs_manager.cc:359] Metadata directory not provided
I20260929 20:50:12.826274 15962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:50:12.826553 15962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:50:12.829413 15962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "275bc00f37b34367a4487c1ca7be4503"
format_stamp: "Formatted at 2026-09-29 20:50:12 on dist-test-slave-blv7"
server_key: "88e62f108539b678fb39c74a5ba4f7cf"
server_key_iv: "3e03b02609822a64d29a6e5a33e23a84"
server_key_version: "encryptionkey@0"
I20260929 20:50:12.830150 15962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "275bc00f37b34367a4487c1ca7be4503"
format_stamp: "Formatted at 2026-09-29 20:50:12 on dist-test-slave-blv7"
server_key: "88e62f108539b678fb39c74a5ba4f7cf"
server_key_iv: "3e03b02609822a64d29a6e5a33e23a84"
server_key_version: "encryptionkey@0"
I20260929 20:50:12.835104 15962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 20:50:12.838479 15977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:12.839885 15962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:50:12.840032 15962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "275bc00f37b34367a4487c1ca7be4503"
format_stamp: "Formatted at 2026-09-29 20:50:12 on dist-test-slave-blv7"
server_key: "88e62f108539b678fb39c74a5ba4f7cf"
server_key_iv: "3e03b02609822a64d29a6e5a33e23a84"
server_key_version: "encryptionkey@0"
I20260929 20:50:12.840217 15962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:50:12.883249 15962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:50:12.887081 15962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:50:12.887385 15962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:50:12.888331 15962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:50:12.889650 15962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:50:12.889727 15962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:12.889846 15962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:50:12.889904 15962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:12.919516 15962 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33593
I20260929 20:50:12.919575 16089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33593 every 8 connection(s)
I20260929 20:50:12.920838 15962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:50:12.931241 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 15962
I20260929 20:50:12.931456 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:50:12.932109 388 external_mini_cluster.cc:1428] Setting key a2cc053aaf139c52d113ed60718edde5
Sep 29 20:50:12 dist-test-slave-blv7 krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:50:12.941707 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:12.923436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50595 (local address 127.0.97.62:42091)
0929 20:50:12.923853 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:12.923858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:12.924793 (+ 935us) server_negotiation.cc:407] Connection header received
0929 20:50:12.926663 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:12.926669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:12.926752 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:12.926847 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:50:12.928431 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:12.929141 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:12.930702 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:12.931311 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:12.931444 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:12.935460 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:50:12.935467 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:50:12.935473 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:50:12.935514 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:50:12.938584 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:12.939436 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:12.939442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:12.939445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:12.939594 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:12.939998 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:12.940003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:12.940005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:12.940320 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:50:12.940455 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:12.941129 (+ 674us) server_negotiation.cc:299] Negotiation successful
0929 20:50:12.941354 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":233,"threads_started":1}
I20260929 20:50:12.942596 16093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:12.923940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.1:50595)
0929 20:50:12.924622 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:12.924665 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:50:12.926380 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:50:12.927127 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:50:12.927135 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:50:12.927668 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:50:12.928254 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:12.928273 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:12.929309 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:12.929313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:12.930559 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:12.930573 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:12.931557 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:12.931564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:12.931799 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:12.932547 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:50:12.932582 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:50:12.935223 (+ 2641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:50:12.938765 (+ 3542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:50:12.938773 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:50:12.938787 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:50:12.939317 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:50:12.939695 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:50:12.939698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:50:12.939700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:50:12.939892 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:50:12.940423 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:50:12.940429 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:50:12.940771 (+ 342us) client_negotiation.cc:769] Sending connection context
0929 20:50:12.940999 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 20:50:12.941409 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":269,"threads_started":1}
I20260929 20:50:12.944128 16090 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42091
I20260929 20:50:12.944454 16090 heartbeater.cc:464] Registering TS with master...
I20260929 20:50:12.945175 16090 heartbeater.cc:511] Master 127.0.97.62:42091 requested a full tablet report, sending...
I20260929 20:50:12.947217 15901 ts_manager.cc:194] Registered new tserver with Master: 275bc00f37b34367a4487c1ca7be4503 (127.0.97.1:33593)
I20260929 20:50:12.948649 15901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50595
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:50:12.979797 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42091
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:50:13.294337 16096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:50:13.294617 16096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:50:13.294741 16096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:50:13.302017 16096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:50:13.302155 16096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:50:13.302310 16096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:50:13.312016 16096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:50:13.313287 16096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:50:13.314524 16096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:50:13.327286 16102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:13.327368 16104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:13.327682 16101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:50:13.328998 16096 server_base.cc:1034] running on GCE node
I20260929 20:50:13.329659 16096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:50:13.330755 16096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:50:13.331943 16096 hybrid_clock.cc:648] HybridClock initialized: now 1790715013331890 us; error 48 us; skew 500 ppm
I20260929 20:50:13.332211 16096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:50:13 dist-test-slave-blv7 krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:50:13.335151 16096 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:50:13.336563 16096 webserver.cc:460] Webserver started at http://127.0.97.2:36571/ using document root <none> and password file <none>
I20260929 20:50:13.337253 16096 fs_manager.cc:359] Metadata directory not provided
I20260929 20:50:13.337355 16096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:50:13.337639 16096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:50:13.340528 16096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63"
format_stamp: "Formatted at 2026-09-29 20:50:13 on dist-test-slave-blv7"
server_key: "3345e601c291499b972316c59af06cfa"
server_key_iv: "bb8b0013e063c7f43f03a82fdf7fbe62"
server_key_version: "encryptionkey@0"
I20260929 20:50:13.341244 16096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63"
format_stamp: "Formatted at 2026-09-29 20:50:13 on dist-test-slave-blv7"
server_key: "3345e601c291499b972316c59af06cfa"
server_key_iv: "bb8b0013e063c7f43f03a82fdf7fbe62"
server_key_version: "encryptionkey@0"
I20260929 20:50:13.346282 16096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 20:50:13.349673 16111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:13.351068 16096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:50:13.351221 16096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63"
format_stamp: "Formatted at 2026-09-29 20:50:13 on dist-test-slave-blv7"
server_key: "3345e601c291499b972316c59af06cfa"
server_key_iv: "bb8b0013e063c7f43f03a82fdf7fbe62"
server_key_version: "encryptionkey@0"
I20260929 20:50:13.351405 16096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:50:13.371084 16096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:50:13.375176 16096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:50:13.375453 16096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:50:13.376425 16096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:50:13.377784 16096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:50:13.377854 16096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:13.377976 16096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:50:13.378032 16096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:13.407359 16096 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39055
I20260929 20:50:13.407425 16223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39055 every 8 connection(s)
I20260929 20:50:13.408563 16096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:50:13.409979 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 16096
I20260929 20:50:13.410147 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:50:13.410557 388 external_mini_cluster.cc:1428] Setting key 196fcc2be8bb63b1bd093cefb0da46d0
Sep 29 20:50:13 dist-test-slave-blv7 krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:50:13.429760 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:13.412203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37341 (local address 127.0.97.62:42091)
0929 20:50:13.412394 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:13.412400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:13.413717 (+ 1317us) server_negotiation.cc:407] Connection header received
0929 20:50:13.415508 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:13.415514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:13.415591 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:13.415685 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:50:13.417070 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:13.418019 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:13.419539 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:13.420163 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:13.420278 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:13.423747 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:50:13.423753 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:50:13.423756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:50:13.423789 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:50:13.426630 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:13.427488 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:13.427492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:13.427494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:13.427615 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:13.428074 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:13.428078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:13.428080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:13.428343 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:50:13.428449 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:13.429325 (+ 876us) server_negotiation.cc:299] Negotiation successful
0929 20:50:13.429506 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:50:13.431214 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:13.412776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.2:37341)
0929 20:50:13.413545 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:13.413589 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:50:13.415269 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:50:13.415888 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:50:13.415896 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:50:13.416378 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:50:13.416920 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:13.416937 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:13.418142 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:13.418145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:13.419402 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:13.419414 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:13.420333 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:13.420337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:13.420514 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:13.421134 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:50:13.421154 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:50:13.423497 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:50:13.426780 (+ 3283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:50:13.426789 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:50:13.426808 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:50:13.427377 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:50:13.427716 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:50:13.427720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:50:13.427724 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:50:13.427963 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:50:13.428462 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:50:13.428469 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:50:13.428916 (+ 447us) client_negotiation.cc:769] Sending connection context
0929 20:50:13.429233 (+ 317us) client_negotiation.cc:240] Negotiation successful
0929 20:50:13.429793 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":317,"threads_started":1}
I20260929 20:50:13.432593 16224 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42091
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:50:13.432924 16224 heartbeater.cc:464] Registering TS with master...
I20260929 20:50:13.433651 16224 heartbeater.cc:511] Master 127.0.97.62:42091 requested a full tablet report, sending...
I20260929 20:50:13.435145 15901 ts_manager.cc:194] Registered new tserver with Master: 4f5d8a998dcb4ed1a8e536e5c40b5d63 (127.0.97.2:39055)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 20:50:13.435986 15901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:50:13.461797 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42091
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:50:13.770823 16229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:50:13.771121 16229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:50:13.771256 16229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:50:13.778358 16229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:50:13.778465 16229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:50:13.778623 16229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:50:13.788594 16229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:50:13.789845 16229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:50:13.791116 16229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:50:13.803783 16237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:13.803805 16235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:13.804021 16234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:50:13.804674 16229 server_base.cc:1034] running on GCE node
I20260929 20:50:13.805312 16229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:50:13.806243 16229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:50:13.807521 16229 hybrid_clock.cc:648] HybridClock initialized: now 1790715013807465 us; error 50 us; skew 500 ppm
I20260929 20:50:13.807827 16229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:50:13 dist-test-slave-blv7 krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:50:13.810832 16229 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:50:13.812265 16229 webserver.cc:460] Webserver started at http://127.0.97.3:34913/ using document root <none> and password file <none>
I20260929 20:50:13.812919 16229 fs_manager.cc:359] Metadata directory not provided
I20260929 20:50:13.813035 16229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:50:13.813339 16229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:50:13.816144 16229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "77d3674ed2f74c4cab409a034357df57"
format_stamp: "Formatted at 2026-09-29 20:50:13 on dist-test-slave-blv7"
server_key: "773e86a27cdbc1b758e2b9087a4bb884"
server_key_iv: "c3c30176bfb36b0de5002503dbf744fe"
server_key_version: "encryptionkey@0"
I20260929 20:50:13.816835 16229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "77d3674ed2f74c4cab409a034357df57"
format_stamp: "Formatted at 2026-09-29 20:50:13 on dist-test-slave-blv7"
server_key: "773e86a27cdbc1b758e2b9087a4bb884"
server_key_iv: "c3c30176bfb36b0de5002503dbf744fe"
server_key_version: "encryptionkey@0"
I20260929 20:50:13.821797 16229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 20:50:13.825127 16244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:13.826447 16229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:50:13.826601 16229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "77d3674ed2f74c4cab409a034357df57"
format_stamp: "Formatted at 2026-09-29 20:50:13 on dist-test-slave-blv7"
server_key: "773e86a27cdbc1b758e2b9087a4bb884"
server_key_iv: "c3c30176bfb36b0de5002503dbf744fe"
server_key_version: "encryptionkey@0"
I20260929 20:50:13.826808 16229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:50:13.841504 16229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:50:13.845445 16229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:50:13.845702 16229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:50:13.846585 16229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:50:13.847875 16229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:50:13.847949 16229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:13.848064 16229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:50:13.848114 16229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:13.878706 16229 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40117
I20260929 20:50:13.878753 16356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40117 every 8 connection(s)
I20260929 20:50:13.880064 16229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:50:13.881609 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 16229
I20260929 20:50:13.881773 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:50:13.882340 388 external_mini_cluster.cc:1428] Setting key 5d14ac8856f1eb9d72c89322506192ae
Sep 29 20:50:13 dist-test-slave-blv7 krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:50:13.906097 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:13.883834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57723 (local address 127.0.97.62:42091)
0929 20:50:13.884033 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:13.884040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:13.884513 (+ 473us) server_negotiation.cc:407] Connection header received
0929 20:50:13.886503 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:13.886508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:13.886588 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:13.886679 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:50:13.888141 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:13.888910 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:13.890612 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:13.891008 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:13.891146 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:13.901308 (+ 10162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:50:13.901312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:50:13.901316 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:50:13.901349 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:50:13.903622 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:13.904240 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:13.904244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:13.904246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:13.904362 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:13.904686 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:13.904689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:13.904691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:13.904937 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:50:13.905042 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:13.905689 (+ 647us) server_negotiation.cc:299] Negotiation successful
0929 20:50:13.905851 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 20:50:13.907123 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:13.883368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.3:57723)
0929 20:50:13.884333 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:13.884389 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 20:50:13.886263 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:50:13.886858 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:50:13.886866 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:50:13.887430 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:50:13.887977 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:13.887995 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:13.889019 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:13.889023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:13.890469 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:13.890482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:13.891228 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:13.891236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:13.891473 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:13.898879 (+ 7406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:50:13.898904 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:50:13.901101 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:50:13.903738 (+ 2637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:50:13.903745 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:50:13.903758 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:50:13.904150 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:50:13.904435 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:50:13.904438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:50:13.904440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:50:13.904609 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:50:13.905021 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:50:13.905055 (+ 34us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:50:13.905377 (+ 322us) client_negotiation.cc:769] Sending connection context
0929 20:50:13.905591 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 20:50:13.905961 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":387,"threads_started":1}
I20260929 20:50:13.908562 16357 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42091
I20260929 20:50:13.908875 16357 heartbeater.cc:464] Registering TS with master...
I20260929 20:50:13.909617 16357 heartbeater.cc:511] Master 127.0.97.62:42091 requested a full tablet report, sending...
I20260929 20:50:13.910852 15901 ts_manager.cc:194] Registered new tserver with Master: 77d3674ed2f74c4cab409a034357df57 (127.0.97.3:40117)
I20260929 20:50:13.911643 15901 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57723
I20260929 20:50:13.921623 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:50:13.938308 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:13.926228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.0.97.62:42091)
0929 20:50:13.926394 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:13.926401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:13.926774 (+ 373us) server_negotiation.cc:407] Connection header received
0929 20:50:13.926935 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:13.926939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:13.926999 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:13.927110 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:50:13.928533 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:13.929235 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:13.930697 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:13.931152 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:13.931263 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:13.933483 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:50:13.933488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:50:13.933491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:50:13.933529 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:50:13.935882 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:13.936481 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:13.936486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:13.936490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:13.936613 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:13.936946 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:13.936950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:13.936954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:13.937278 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:50:13.937494 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:13.937806 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 20:50:13.938010 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 20:50:13.941933 15901 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:50:13.948547 15901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:50:13.951804 16090 heartbeater.cc:503] Master 127.0.97.62:42091 was elected leader, sending a full tablet report...
I20260929 20:50:13.982057 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:13.960207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45436 (local address 127.0.97.3:40117)
0929 20:50:13.970573 (+ 10366us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:13.970584 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:13.973151 (+ 2567us) server_negotiation.cc:407] Connection header received
0929 20:50:13.973281 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:13.973289 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:13.973562 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:13.973745 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:50:13.975519 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:13.977046 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:13.979700 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:13.980617 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:13.980733 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:13.980910 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:13.981531 (+ 621us) server_negotiation.cc:299] Negotiation successful
0929 20:50:13.981701 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10157,"thread_start_us":9987,"threads_started":1}
I20260929 20:50:13.982101 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:13.960079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40117 (local address 127.0.0.1:45436)
0929 20:50:13.972687 (+ 12608us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:13.972732 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:50:13.972948 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:50:13.974820 (+ 1872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:50:13.974826 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:50:13.974855 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:50:13.975318 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:13.975330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:13.977187 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:13.977192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:13.979507 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:13.979517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:13.981011 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:13.981017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:13.981254 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:13.981318 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 20:50:13.981447 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 20:50:13.981645 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12323,"mutex_wait_us":122,"spinlock_wait_cycles":896,"thread_start_us":14361,"threads_started":1}
I20260929 20:50:13.982450 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:13.960895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43936 (local address 127.0.97.1:33593)
0929 20:50:13.974418 (+ 13523us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:13.974425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:13.974452 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:50:13.974540 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:13.974547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:13.974868 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:13.975134 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:50:13.975716 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:13.976807 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:13.979655 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:13.981021 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:13.981150 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:13.981363 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:13.981620 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 20:50:13.981934 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13364,"thread_start_us":13188,"threads_started":1}
I20260929 20:50:13.982879 16373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:13.960153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33593 (local address 127.0.0.1:43936)
0929 20:50:13.973731 (+ 13578us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:13.973750 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:50:13.973870 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:50:13.975070 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:50:13.975075 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:50:13.975096 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:50:13.975505 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:13.975515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:13.976928 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:13.976933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:13.979503 (+ 2570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:13.979517 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:13.981197 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:13.981203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:13.981349 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:13.981373 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:50:13.981447 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 20:50:13.982566 (+ 1119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13423,"mutex_wait_us":1007,"thread_start_us":12098,"threads_started":1}
I20260929 20:50:13.983594 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:13.961353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39055 (local address 127.0.0.1:41956)
0929 20:50:13.974686 (+ 13333us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:13.974753 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 20:50:13.974892 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:50:13.975558 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:50:13.975562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:50:13.975586 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:50:13.975919 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:13.975930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:13.977564 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:13.977568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:13.979434 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:13.979449 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:13.981497 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:13.981503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:13.982981 (+ 1478us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:13.983012 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:50:13.983132 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 20:50:13.983253 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13225,"mutex_wait_us":2,"spinlock_wait_cycles":3328,"thread_start_us":12616,"threads_started":1}
I20260929 20:50:13.983652 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:13.961554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41956 (local address 127.0.97.2:39055)
0929 20:50:13.973303 (+ 11749us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:13.973312 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:13.975059 (+ 1747us) server_negotiation.cc:407] Connection header received
0929 20:50:13.975161 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:13.975168 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:13.975388 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:13.975568 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:50:13.976075 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:13.977409 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:13.979578 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:13.981037 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:13.981150 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:13.981332 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:13.983284 (+ 1952us) server_negotiation.cc:299] Negotiation successful
0929 20:50:13.983417 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11602,"thread_start_us":11493,"threads_started":1}
I20260929 20:50:13.986371 16292 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1fe65b046c47678af7553fab878fc5 (DEFAULT_TABLE table=test-table [id=8e5e8ace3b51488db959e2d46771aa84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:50:13.986670 16159 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1fe65b046c47678af7553fab878fc5 (DEFAULT_TABLE table=test-table [id=8e5e8ace3b51488db959e2d46771aa84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:50:13.986595 16025 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1fe65b046c47678af7553fab878fc5 (DEFAULT_TABLE table=test-table [id=8e5e8ace3b51488db959e2d46771aa84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:50:13.987761 16292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a1fe65b046c47678af7553fab878fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:50:13.987927 16159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a1fe65b046c47678af7553fab878fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:50:13.988096 16025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a1fe65b046c47678af7553fab878fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:50:13.998220 16375 tablet_bootstrap.cc:492] T 6a1fe65b046c47678af7553fab878fc5 P 77d3674ed2f74c4cab409a034357df57: Bootstrap starting.
I20260929 20:50:14.001775 16375 tablet_bootstrap.cc:654] T 6a1fe65b046c47678af7553fab878fc5 P 77d3674ed2f74c4cab409a034357df57: Neither blocks nor log segments found. Creating new log.
I20260929 20:50:14.002753 16375 log.cc:824] T 6a1fe65b046c47678af7553fab878fc5 P 77d3674ed2f74c4cab409a034357df57: Log is configured to *not* fsync() on all Append() calls
I20260929 20:50:14.005586 16375 tablet_bootstrap.cc:492] T 6a1fe65b046c47678af7553fab878fc5 P 77d3674ed2f74c4cab409a034357df57: No bootstrap required, opened a new log
I20260929 20:50:14.005859 16375 ts_tablet_manager.cc:1397] T 6a1fe65b046c47678af7553fab878fc5 P 77d3674ed2f74c4cab409a034357df57: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 20:50:14.011101 16377 tablet_bootstrap.cc:492] T 6a1fe65b046c47678af7553fab878fc5 P 275bc00f37b34367a4487c1ca7be4503: Bootstrap starting.
I20260929 20:50:14.014655 16377 tablet_bootstrap.cc:654] T 6a1fe65b046c47678af7553fab878fc5 P 275bc00f37b34367a4487c1ca7be4503: Neither blocks nor log segments found. Creating new log.
I20260929 20:50:14.015957 16377 log.cc:824] T 6a1fe65b046c47678af7553fab878fc5 P 275bc00f37b34367a4487c1ca7be4503: Log is configured to *not* fsync() on all Append() calls
I20260929 20:50:14.017205 16376 tablet_bootstrap.cc:492] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: Bootstrap starting.
I20260929 20:50:14.021747 16377 tablet_bootstrap.cc:492] T 6a1fe65b046c47678af7553fab878fc5 P 275bc00f37b34367a4487c1ca7be4503: No bootstrap required, opened a new log
I20260929 20:50:14.022044 16377 ts_tablet_manager.cc:1397] T 6a1fe65b046c47678af7553fab878fc5 P 275bc00f37b34367a4487c1ca7be4503: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260929 20:50:14.022019 16375 raft_consensus.cc:348] T 6a1fe65b046c47678af7553fab878fc5 P 77d3674ed2f74c4cab409a034357df57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d3674ed2f74c4cab409a034357df57" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40117 } } peers { permanent_uuid: "275bc00f37b34367a4487c1ca7be4503" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39055 } }
I20260929 20:50:14.022488 16375 raft_consensus.cc:374] T 6a1fe65b046c47678af7553fab878fc5 P 77d3674ed2f74c4cab409a034357df57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:50:14.022596 16375 raft_consensus.cc:729] T 6a1fe65b046c47678af7553fab878fc5 P 77d3674ed2f74c4cab409a034357df57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77d3674ed2f74c4cab409a034357df57, State: Initialized, Role: FOLLOWER
I20260929 20:50:14.023267 16375 consensus_queue.cc:260] T 6a1fe65b046c47678af7553fab878fc5 P 77d3674ed2f74c4cab409a034357df57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d3674ed2f74c4cab409a034357df57" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40117 } } peers { permanent_uuid: "275bc00f37b34367a4487c1ca7be4503" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39055 } }
I20260929 20:50:14.024574 16376 tablet_bootstrap.cc:654] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: Neither blocks nor log segments found. Creating new log.
I20260929 20:50:14.026026 16376 log.cc:824] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: Log is configured to *not* fsync() on all Append() calls
I20260929 20:50:14.028309 16357 heartbeater.cc:503] Master 127.0.97.62:42091 was elected leader, sending a full tablet report...
I20260929 20:50:14.028916 16375 ts_tablet_manager.cc:1428] T 6a1fe65b046c47678af7553fab878fc5 P 77d3674ed2f74c4cab409a034357df57: Time spent starting tablet: real 0.023s user 0.014s sys 0.003s
I20260929 20:50:14.032675 16377 raft_consensus.cc:348] T 6a1fe65b046c47678af7553fab878fc5 P 275bc00f37b34367a4487c1ca7be4503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d3674ed2f74c4cab409a034357df57" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40117 } } peers { permanent_uuid: "275bc00f37b34367a4487c1ca7be4503" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39055 } }
I20260929 20:50:14.033114 16377 raft_consensus.cc:374] T 6a1fe65b046c47678af7553fab878fc5 P 275bc00f37b34367a4487c1ca7be4503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:50:14.033202 16377 raft_consensus.cc:729] T 6a1fe65b046c47678af7553fab878fc5 P 275bc00f37b34367a4487c1ca7be4503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275bc00f37b34367a4487c1ca7be4503, State: Initialized, Role: FOLLOWER
I20260929 20:50:14.033685 16376 tablet_bootstrap.cc:492] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: No bootstrap required, opened a new log
I20260929 20:50:14.033958 16376 ts_tablet_manager.cc:1397] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.000s
I20260929 20:50:14.033905 16377 consensus_queue.cc:260] T 6a1fe65b046c47678af7553fab878fc5 P 275bc00f37b34367a4487c1ca7be4503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d3674ed2f74c4cab409a034357df57" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40117 } } peers { permanent_uuid: "275bc00f37b34367a4487c1ca7be4503" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39055 } }
I20260929 20:50:14.040427 16377 ts_tablet_manager.cc:1428] T 6a1fe65b046c47678af7553fab878fc5 P 275bc00f37b34367a4487c1ca7be4503: Time spent starting tablet: real 0.018s user 0.019s sys 0.000s
I20260929 20:50:14.044446 16376 raft_consensus.cc:348] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d3674ed2f74c4cab409a034357df57" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40117 } } peers { permanent_uuid: "275bc00f37b34367a4487c1ca7be4503" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39055 } }
I20260929 20:50:14.044932 16376 raft_consensus.cc:374] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:50:14.045025 16376 raft_consensus.cc:729] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5d8a998dcb4ed1a8e536e5c40b5d63, State: Initialized, Role: FOLLOWER
I20260929 20:50:14.045683 16376 consensus_queue.cc:260] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d3674ed2f74c4cab409a034357df57" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40117 } } peers { permanent_uuid: "275bc00f37b34367a4487c1ca7be4503" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39055 } }
I20260929 20:50:14.046741 16224 heartbeater.cc:503] Master 127.0.97.62:42091 was elected leader, sending a full tablet report...
I20260929 20:50:14.047545 16376 ts_tablet_manager.cc:1428] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260929 20:50:14.131631 16358 tablet.cc:2369] T 6a1fe65b046c47678af7553fab878fc5 P 77d3674ed2f74c4cab409a034357df57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:50:14.160391 16225 tablet.cc:2369] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:50:14.172400 16383 raft_consensus.cc:482] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:50:14.172696 16383 raft_consensus.cc:504] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d3674ed2f74c4cab409a034357df57" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40117 } } peers { permanent_uuid: "275bc00f37b34367a4487c1ca7be4503" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39055 } }
W20260929 20:50:14.173204 16091 tablet.cc:2369] T 6a1fe65b046c47678af7553fab878fc5 P 275bc00f37b34367a4487c1ca7be4503: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:50:14.174451 16383 leader_election.cc:288] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77d3674ed2f74c4cab409a034357df57 (127.0.97.3:40117), 275bc00f37b34367a4487c1ca7be4503 (127.0.97.1:33593)
I20260929 20:50:14.181003 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:14.174872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57021 (local address 127.0.97.1:33593)
0929 20:50:14.175092 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:14.175098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:14.175770 (+ 672us) server_negotiation.cc:407] Connection header received
0929 20:50:14.175976 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:14.175980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:14.176038 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:14.176127 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:50:14.176647 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:14.177382 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:14.179269 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:14.180103 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:14.180206 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:14.180289 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:14.180643 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 20:50:14.180771 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:50:14.181036 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:14.174770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45801 (local address 127.0.97.3:40117)
0929 20:50:14.174965 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:14.174972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:14.175753 (+ 781us) server_negotiation.cc:407] Connection header received
0929 20:50:14.175935 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:14.175940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:14.176007 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:14.176071 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:50:14.176808 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:14.177602 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:14.179416 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:14.180205 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:14.180327 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:14.180422 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:14.180702 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 20:50:14.180823 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:50:14.181219 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:14.174739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40117 (local address 127.0.97.2:45801)
0929 20:50:14.175637 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:14.175657 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:50:14.175795 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:50:14.176158 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:50:14.176162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:50:14.176185 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:50:14.176637 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:14.176649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:14.177744 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:14.177747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:14.179275 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:14.179289 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:14.180418 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:14.180423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:14.180548 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:14.180575 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 20:50:14.180664 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 20:50:14.180853 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":3,"thread_start_us":438,"threads_started":1}
I20260929 20:50:14.181466 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:14.174881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33593 (local address 127.0.97.2:57021)
0929 20:50:14.175661 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:14.175675 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:50:14.175847 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:50:14.176147 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:50:14.176152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:50:14.176174 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:50:14.176483 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:14.176492 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:14.177514 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:14.177520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:14.179116 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:14.179129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:14.180320 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:14.180326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:14.180486 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:14.180513 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 20:50:14.180586 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 20:50:14.180691 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"spinlock_wait_cycles":24832,"thread_start_us":463,"threads_started":1}
I20260929 20:50:14.182214 16312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1fe65b046c47678af7553fab878fc5" candidate_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77d3674ed2f74c4cab409a034357df57" is_pre_election: true
I20260929 20:50:14.182314 16045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1fe65b046c47678af7553fab878fc5" candidate_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275bc00f37b34367a4487c1ca7be4503" is_pre_election: true
I20260929 20:50:14.182711 16312 raft_consensus.cc:2454] T 6a1fe65b046c47678af7553fab878fc5 P 77d3674ed2f74c4cab409a034357df57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5d8a998dcb4ed1a8e536e5c40b5d63 in term 0.
I20260929 20:50:14.182780 16045 raft_consensus.cc:2454] T 6a1fe65b046c47678af7553fab878fc5 P 275bc00f37b34367a4487c1ca7be4503 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5d8a998dcb4ed1a8e536e5c40b5d63 in term 0.
I20260929 20:50:14.183312 16114 leader_election.cc:302] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f5d8a998dcb4ed1a8e536e5c40b5d63, 77d3674ed2f74c4cab409a034357df57; no voters:
I20260929 20:50:14.183653 16383 raft_consensus.cc:2781] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:50:14.183799 16383 raft_consensus.cc:482] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:50:14.183897 16383 raft_consensus.cc:3037] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:50:14.185945 16383 raft_consensus.cc:504] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d3674ed2f74c4cab409a034357df57" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40117 } } peers { permanent_uuid: "275bc00f37b34367a4487c1ca7be4503" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39055 } }
I20260929 20:50:14.186535 16383 leader_election.cc:288] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [CANDIDATE]: Term 1 election: Requested vote from peers 77d3674ed2f74c4cab409a034357df57 (127.0.97.3:40117), 275bc00f37b34367a4487c1ca7be4503 (127.0.97.1:33593)
I20260929 20:50:14.186896 16312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1fe65b046c47678af7553fab878fc5" candidate_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77d3674ed2f74c4cab409a034357df57"
I20260929 20:50:14.186976 16045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1fe65b046c47678af7553fab878fc5" candidate_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275bc00f37b34367a4487c1ca7be4503"
I20260929 20:50:14.187096 16312 raft_consensus.cc:3037] T 6a1fe65b046c47678af7553fab878fc5 P 77d3674ed2f74c4cab409a034357df57 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:50:14.187201 16045 raft_consensus.cc:3037] T 6a1fe65b046c47678af7553fab878fc5 P 275bc00f37b34367a4487c1ca7be4503 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:50:14.189240 16045 raft_consensus.cc:2454] T 6a1fe65b046c47678af7553fab878fc5 P 275bc00f37b34367a4487c1ca7be4503 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5d8a998dcb4ed1a8e536e5c40b5d63 in term 1.
I20260929 20:50:14.189252 16312 raft_consensus.cc:2454] T 6a1fe65b046c47678af7553fab878fc5 P 77d3674ed2f74c4cab409a034357df57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5d8a998dcb4ed1a8e536e5c40b5d63 in term 1.
I20260929 20:50:14.189595 16115 leader_election.cc:302] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 275bc00f37b34367a4487c1ca7be4503, 4f5d8a998dcb4ed1a8e536e5c40b5d63; no voters:
I20260929 20:50:14.189851 16383 raft_consensus.cc:2781] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:50:14.190256 16383 raft_consensus.cc:686] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [term 1 LEADER]: Becoming Leader. State: Replica: 4f5d8a998dcb4ed1a8e536e5c40b5d63, State: Running, Role: LEADER
I20260929 20:50:14.190688 16383 consensus_queue.cc:237] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d3674ed2f74c4cab409a034357df57" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40117 } } peers { permanent_uuid: "275bc00f37b34367a4487c1ca7be4503" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39055 } }
I20260929 20:50:14.194998 15901 catalog_manager.cc:5495] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f5d8a998dcb4ed1a8e536e5c40b5d63 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d3674ed2f74c4cab409a034357df57" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "275bc00f37b34367a4487c1ca7be4503" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39055 } health_report { overall_health: HEALTHY } } }
I20260929 20:50:14.218849 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:14.213593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41966 (local address 127.0.97.2:39055)
0929 20:50:14.213759 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:14.213763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:14.213803 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:50:14.213889 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:14.213893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:14.213967 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:14.214126 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:50:14.214440 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:14.215261 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:14.216811 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:14.217232 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:14.217333 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:14.217785 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:50:14.218342 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:50:14.218449 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:14.218560 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 20:50:14.218634 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 20:50:14.226903 16045 raft_consensus.cc:1261] T 6a1fe65b046c47678af7553fab878fc5 P 275bc00f37b34367a4487c1ca7be4503 [term 1 FOLLOWER]: Refusing update from remote peer 4f5d8a998dcb4ed1a8e536e5c40b5d63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:50:14.226903 16312 raft_consensus.cc:1261] T 6a1fe65b046c47678af7553fab878fc5 P 77d3674ed2f74c4cab409a034357df57 [term 1 FOLLOWER]: Refusing update from remote peer 4f5d8a998dcb4ed1a8e536e5c40b5d63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:50:14.227880 16386 consensus_queue.cc:1035] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "275bc00f37b34367a4487c1ca7be4503" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:50:14.228214 16383 consensus_queue.cc:1035] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77d3674ed2f74c4cab409a034357df57" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:50:14.241042 16391 mvcc.cc:205] Tried to move back new op lower bound from 7334768698263670784 to 7334768698128785408. Current Snapshot: MvccSnapshot[applied={T|T < 7334768698263670784}]
I20260929 20:50:14.254490 15900 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:50:14.254798 15900 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:50:14.258533 15900 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379: Sending DeleteTablet for 3 replicas of tablet 6a1fe65b046c47678af7553fab878fc5
I20260929 20:50:14.259727 16159 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a1fe65b046c47678af7553fab878fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:50:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41956
I20260929 20:50:14.259718 16025 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a1fe65b046c47678af7553fab878fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:50:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43936
I20260929 20:50:14.259752 16292 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a1fe65b046c47678af7553fab878fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:50:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45436
I20260929 20:50:14.264020 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 15962
I20260929 20:50:14.272282 16401 tablet_replica.cc:330] T 6a1fe65b046c47678af7553fab878fc5 P 77d3674ed2f74c4cab409a034357df57: stopping tablet replica
I20260929 20:50:14.272789 16401 raft_consensus.cc:2229] T 6a1fe65b046c47678af7553fab878fc5 P 77d3674ed2f74c4cab409a034357df57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:50:14.273386 16401 raft_consensus.cc:2258] T 6a1fe65b046c47678af7553fab878fc5 P 77d3674ed2f74c4cab409a034357df57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:50:14.276310 16399 tablet_replica.cc:330] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: stopping tablet replica
I20260929 20:50:14.276871 16399 raft_consensus.cc:2229] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:50:14.277572 16401 ts_tablet_manager.cc:1905] T 6a1fe65b046c47678af7553fab878fc5 P 77d3674ed2f74c4cab409a034357df57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:50:14.277776 16399 raft_consensus.cc:2258] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:50:14.278678 16400 tablet_replica.cc:330] T 6a1fe65b046c47678af7553fab878fc5 P 275bc00f37b34367a4487c1ca7be4503: stopping tablet replica
I20260929 20:50:14.279186 16400 raft_consensus.cc:2229] T 6a1fe65b046c47678af7553fab878fc5 P 275bc00f37b34367a4487c1ca7be4503 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:50:14.279767 16400 raft_consensus.cc:2258] T 6a1fe65b046c47678af7553fab878fc5 P 275bc00f37b34367a4487c1ca7be4503 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:50:14.280118 16399 ts_tablet_manager.cc:1905] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:50:14.284584 16400 ts_tablet_manager.cc:1905] T 6a1fe65b046c47678af7553fab878fc5 P 275bc00f37b34367a4487c1ca7be4503: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:50:14.285135 16401 ts_tablet_manager.cc:1918] T 6a1fe65b046c47678af7553fab878fc5 P 77d3674ed2f74c4cab409a034357df57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:50:14.285307 16401 log.cc:1196] T 6a1fe65b046c47678af7553fab878fc5 P 77d3674ed2f74c4cab409a034357df57: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/6a1fe65b046c47678af7553fab878fc5
I20260929 20:50:14.285808 16401 ts_tablet_manager.cc:1939] T 6a1fe65b046c47678af7553fab878fc5 P 77d3674ed2f74c4cab409a034357df57: Deleting consensus metadata
I20260929 20:50:14.287650 15887 catalog_manager.cc:4841] TS 77d3674ed2f74c4cab409a034357df57 (127.0.97.3:40117): tablet 6a1fe65b046c47678af7553fab878fc5 (table test-table [id=8e5e8ace3b51488db959e2d46771aa84]) successfully deleted
I20260929 20:50:14.287730 16399 ts_tablet_manager.cc:1918] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:50:14.287883 16399 log.cc:1196] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/6a1fe65b046c47678af7553fab878fc5
I20260929 20:50:14.288425 16399 ts_tablet_manager.cc:1939] T 6a1fe65b046c47678af7553fab878fc5 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: Deleting consensus metadata
I20260929 20:50:14.290114 15886 catalog_manager.cc:4841] TS 4f5d8a998dcb4ed1a8e536e5c40b5d63 (127.0.97.2:39055): tablet 6a1fe65b046c47678af7553fab878fc5 (table test-table [id=8e5e8ace3b51488db959e2d46771aa84]) successfully deleted
I20260929 20:50:14.291203 16400 ts_tablet_manager.cc:1918] T 6a1fe65b046c47678af7553fab878fc5 P 275bc00f37b34367a4487c1ca7be4503: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:50:14.291356 16400 log.cc:1196] T 6a1fe65b046c47678af7553fab878fc5 P 275bc00f37b34367a4487c1ca7be4503: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/6a1fe65b046c47678af7553fab878fc5
I20260929 20:50:14.291834 16400 ts_tablet_manager.cc:1939] T 6a1fe65b046c47678af7553fab878fc5 P 275bc00f37b34367a4487c1ca7be4503: Deleting consensus metadata
Sep 29 20:50:14 dist-test-slave-blv7 krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:50:14.293432 15888 catalog_manager.cc:4841] TS 275bc00f37b34367a4487c1ca7be4503 (127.0.97.1:33593): tablet 6a1fe65b046c47678af7553fab878fc5 (table test-table [id=8e5e8ace3b51488db959e2d46771aa84]) successfully deleted
I20260929 20:50:14.299768 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:14.265269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43938 (local address 127.0.97.1:33593)
0929 20:50:14.265471 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:14.265477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:14.275685 (+ 10208us) server_negotiation.cc:407] Connection header received
0929 20:50:14.275868 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:14.275875 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:14.276003 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:14.276112 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:50:14.278137 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:14.279344 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:14.283126 (+ 3782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:14.283743 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:14.283841 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:14.293160 (+ 9319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:50:14.293177 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:50:14.293197 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:50:14.293241 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:50:14.296831 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:14.297546 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:14.297556 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:14.297560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:14.297728 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:14.298063 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:14.298068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:14.298071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:14.298525 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:50:14.298831 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:14.299156 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 20:50:14.299377 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:50:14.300405 16085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:50:14.389401 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 15962
I20260929 20:50:14.404520 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 16096
Sep 29 20:50:14 dist-test-slave-blv7 krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:50:14.423889 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:14.405531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41972 (local address 127.0.97.2:39055)
0929 20:50:14.405692 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:14.405697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:14.405717 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:50:14.405773 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:14.405777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:14.405839 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:14.405918 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:50:14.407625 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:14.408971 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:14.410922 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:14.411594 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:14.411724 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:14.416936 (+ 5212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:50:14.416949 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:50:14.416966 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:50:14.417013 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:50:14.420804 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:14.421588 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:14.421598 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:14.421601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:14.421775 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:14.422183 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:14.422189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:14.422192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:14.422614 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:50:14.422942 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:14.423313 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 20:50:14.423526 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:50:14.424680 16219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:50:14.541766 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 16096
I20260929 20:50:14.556370 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 16229
Sep 29 20:50:14 dist-test-slave-blv7 krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:50:14.570981 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:14.557050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45446 (local address 127.0.97.3:40117)
0929 20:50:14.557226 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:14.557232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:14.557309 (+ 77us) server_negotiation.cc:407] Connection header received
0929 20:50:14.557486 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:14.557492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:14.557576 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:14.557656 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:50:14.558941 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:14.559691 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:14.561057 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:14.561439 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:14.561543 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:14.565857 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:50:14.565870 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:50:14.565885 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:50:14.565916 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:50:14.568433 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:14.569042 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:14.569050 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:14.569052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:14.569188 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:14.569519 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:14.569523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:14.569525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:14.569852 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:50:14.570112 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:14.570445 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 20:50:14.570657 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:50:14.571683 16352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:50:14.659322 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 16229
I20260929 20:50:14.673954 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 15869
I20260929 20:50:14.675084 15931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:50:14.782624 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 15869
I20260929 20:50:34.799974 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42091
--webserver_interface=127.0.97.62
--webserver_port=38015
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:50:35.115818 16407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:50:35.116098 16407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:50:35.116192 16407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:50:35.123302 16407 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 20:50:35.123417 16407 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 20:50:35.123456 16407 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 20:50:35.123507 16407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:50:35.123564 16407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:50:35.123648 16407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:50:35.123709 16407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:50:35.123775 16407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:50:35.133759 16407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:50:35.134984 16407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:50:35.136299 16407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:50:35.157263 16416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:35.157279 16414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:35.157263 16413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:50:35.158069 16407 server_base.cc:1034] running on GCE node
I20260929 20:50:35.158913 16407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:50:35.160450 16407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:50:35.161669 16407 hybrid_clock.cc:648] HybridClock initialized: now 1790715035161599 us; error 44 us; skew 500 ppm
I20260929 20:50:35.162053 16407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:50:35 dist-test-slave-blv7 krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:50:35.165407 16407 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:50:35.167047 16407 webserver.cc:460] Webserver started at http://127.0.97.62:38015/ using document root <none> and password file <none>
I20260929 20:50:35.167837 16407 fs_manager.cc:359] Metadata directory not provided
I20260929 20:50:35.167963 16407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:50:35.173965 16407 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 20:50:35.176858 16423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:35.178200 16407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:50:35.178365 16407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "cab36775e31448d180fd3daf0675b379"
format_stamp: "Formatted at 2026-09-29 20:50:12 on dist-test-slave-blv7"
server_key: "e8a34a1b6a61a012417edb04677f2da4"
server_key_iv: "0331bd94cabc8f407f23bba738328061"
server_key_version: "encryptionkey@0"
I20260929 20:50:35.179459 16407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:50:35.207263 16407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:50:35.211515 16407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:50:35.211796 16407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:50:35.229924 16407 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42091
I20260929 20:50:35.229969 16474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42091 every 8 connection(s)
I20260929 20:50:35.231510 16407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:50:35.238974 16475 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:50:35.241349 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 16407
I20260929 20:50:35.242167 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33593
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42091
--webserver_port=41535
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:50:35.242643 16475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379: Bootstrap starting.
I20260929 20:50:35.263242 16475 log.cc:824] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379: Log is configured to *not* fsync() on all Append() calls
I20260929 20:50:35.276517 16475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:50:35.277102 16475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379: Bootstrap complete.
I20260929 20:50:35.286826 16475 raft_consensus.cc:348] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab36775e31448d180fd3daf0675b379" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42091 } }
I20260929 20:50:35.287811 16475 raft_consensus.cc:729] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab36775e31448d180fd3daf0675b379, State: Initialized, Role: FOLLOWER
I20260929 20:50:35.288465 16475 consensus_queue.cc:260] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab36775e31448d180fd3daf0675b379" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42091 } }
I20260929 20:50:35.288731 16475 raft_consensus.cc:388] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:50:35.288858 16475 raft_consensus.cc:482] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:50:35.289034 16475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:50:35.293162 16475 raft_consensus.cc:504] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab36775e31448d180fd3daf0675b379" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42091 } }
I20260929 20:50:35.293704 16475 leader_election.cc:302] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cab36775e31448d180fd3daf0675b379; no voters:
I20260929 20:50:35.294368 16475 leader_election.cc:288] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 20:50:35.294518 16479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:50:35.295605 16479 raft_consensus.cc:686] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 2 LEADER]: Becoming Leader. State: Replica: cab36775e31448d180fd3daf0675b379, State: Running, Role: LEADER
I20260929 20:50:35.296290 16479 consensus_queue.cc:237] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab36775e31448d180fd3daf0675b379" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42091 } }
I20260929 20:50:35.296761 16475 sys_catalog.cc:564] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:50:35.299134 16480 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cab36775e31448d180fd3daf0675b379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab36775e31448d180fd3daf0675b379" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42091 } } }
I20260929 20:50:35.299551 16480 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [sys.catalog]: This master's current role is: LEADER
I20260929 20:50:35.299129 16481 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cab36775e31448d180fd3daf0675b379. Latest consensus state: current_term: 2 leader_uuid: "cab36775e31448d180fd3daf0675b379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab36775e31448d180fd3daf0675b379" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42091 } } }
I20260929 20:50:35.299903 16481 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [sys.catalog]: This master's current role is: LEADER
I20260929 20:50:35.300632 16485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:50:35.311483 16485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:50:35.313119 16485 catalog_manager.cc:1259] Loaded cluster ID: 2108cd2f26b5401bab680f8c3f0ae652
I20260929 20:50:35.313251 16485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:50:35.317380 16485 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:50:35.319061 16485 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379: Loaded TSK: 0
I20260929 20:50:35.365669 16485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379: Generated new TSK 1
I20260929 20:50:35.366145 16485 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 20:50:35.588527 16477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:50:35.588804 16477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:50:35.588934 16477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:50:35.596056 16477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:50:35.596174 16477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:50:35.596325 16477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:50:35.605883 16477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41535
--tserver_master_addrs=127.0.97.62:42091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:50:35.607170 16477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:50:35.608395 16477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:50:35.622673 16503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:35.622859 16502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:35.623041 16505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:50:35.624738 16477 server_base.cc:1034] running on GCE node
I20260929 20:50:35.625427 16477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:50:35.626442 16477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:50:35.627668 16477 hybrid_clock.cc:648] HybridClock initialized: now 1790715035627604 us; error 52 us; skew 500 ppm
I20260929 20:50:35.627997 16477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:50:35 dist-test-slave-blv7 krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:50:35.631130 16477 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:50:35.632624 16477 webserver.cc:460] Webserver started at http://127.0.97.1:41535/ using document root <none> and password file <none>
I20260929 20:50:35.633354 16477 fs_manager.cc:359] Metadata directory not provided
I20260929 20:50:35.633478 16477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:50:35.638850 16477 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:50:35.641508 16512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:35.642828 16477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:50:35.642975 16477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "275bc00f37b34367a4487c1ca7be4503"
format_stamp: "Formatted at 2026-09-29 20:50:12 on dist-test-slave-blv7"
server_key: "88e62f108539b678fb39c74a5ba4f7cf"
server_key_iv: "3e03b02609822a64d29a6e5a33e23a84"
server_key_version: "encryptionkey@0"
I20260929 20:50:35.643953 16477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:50:35.662243 16477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:50:35.666386 16477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:50:35.666625 16477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:50:35.667647 16477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:50:35.669762 16477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:50:35.669852 16477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:35.669946 16477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:50:35.670017 16477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:35.708099 16477 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33593
I20260929 20:50:35.708159 16624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33593 every 8 connection(s)
I20260929 20:50:35.709558 16477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:50:35.714020 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 16477
I20260929 20:50:35.714701 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:39055
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42091
--webserver_port=36571
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:50:35 dist-test-slave-blv7 krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:50:35.734797 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:35.712515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54951 (local address 127.0.97.62:42091)
0929 20:50:35.713099 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:35.713112 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:35.714231 (+ 1119us) server_negotiation.cc:407] Connection header received
0929 20:50:35.716355 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:35.716373 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:35.716879 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:35.717300 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:50:35.718831 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:35.720580 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:35.722404 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:35.723176 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:35.723315 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:35.727351 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:50:35.727360 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:50:35.727376 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:50:35.727415 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:50:35.730130 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:35.730869 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:35.730877 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:35.730883 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:35.731078 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:35.731626 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:35.731630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:35.731633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:35.732131 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:50:35.732306 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:35.733410 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0929 20:50:35.733774 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":254,"threads_started":1}
I20260929 20:50:35.735214 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:35.712839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.1:54951)
0929 20:50:35.713928 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:35.713986 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 20:50:35.715788 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:50:35.717444 (+ 1656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:50:35.717452 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:50:35.718034 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:50:35.718614 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:35.718634 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:35.720767 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:35.720771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:35.722266 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:35.722282 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:35.723426 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:35.723433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:35.723690 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:35.724412 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:50:35.724439 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:50:35.727093 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:50:35.730274 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:50:35.730284 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:50:35.730300 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:50:35.730730 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:50:35.731208 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:50:35.731214 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:50:35.731218 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:50:35.731475 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:50:35.732300 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:50:35.732309 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:50:35.732768 (+ 459us) client_negotiation.cc:769] Sending connection context
0929 20:50:35.733202 (+ 434us) client_negotiation.cc:240] Negotiation successful
0929 20:50:35.733808 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":394,"threads_started":1}
I20260929 20:50:35.738261 16625 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42091
I20260929 20:50:35.738574 16625 heartbeater.cc:464] Registering TS with master...
I20260929 20:50:35.739331 16625 heartbeater.cc:511] Master 127.0.97.62:42091 requested a full tablet report, sending...
I20260929 20:50:35.741668 16440 ts_manager.cc:194] Registered new tserver with Master: 275bc00f37b34367a4487c1ca7be4503 (127.0.97.1:33593)
I20260929 20:50:35.743237 16440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54951
W20260929 20:50:36.042796 16629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:50:36.043100 16629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:50:36.043252 16629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:50:36.050549 16629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:50:36.050662 16629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:50:36.050813 16629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:50:36.060537 16629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:39055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36571
--tserver_master_addrs=127.0.97.62:42091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:50:36.061821 16629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:50:36.063218 16629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:50:36.075476 16634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:36.075508 16637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:36.075538 16635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:50:36.076157 16629 server_base.cc:1034] running on GCE node
I20260929 20:50:36.076759 16629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:50:36.077685 16629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:50:36.078894 16629 hybrid_clock.cc:648] HybridClock initialized: now 1790715036078835 us; error 51 us; skew 500 ppm
I20260929 20:50:36.079218 16629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:50:36 dist-test-slave-blv7 krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:50:36.082065 16629 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:50:36.083630 16629 webserver.cc:460] Webserver started at http://127.0.97.2:36571/ using document root <none> and password file <none>
I20260929 20:50:36.084573 16629 fs_manager.cc:359] Metadata directory not provided
I20260929 20:50:36.084693 16629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:50:36.090065 16629 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 20:50:36.092862 16644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:36.094236 16629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:50:36.094399 16629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63"
format_stamp: "Formatted at 2026-09-29 20:50:13 on dist-test-slave-blv7"
server_key: "3345e601c291499b972316c59af06cfa"
server_key_iv: "bb8b0013e063c7f43f03a82fdf7fbe62"
server_key_version: "encryptionkey@0"
I20260929 20:50:36.095345 16629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:50:36.126626 16629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:50:36.130893 16629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:50:36.131163 16629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:50:36.132071 16629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:50:36.133322 16629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:50:36.133404 16629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:36.133478 16629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:50:36.133531 16629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:36.163302 16629 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39055
I20260929 20:50:36.163405 16756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39055 every 8 connection(s)
I20260929 20:50:36.164672 16629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:50:36.169196 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 16629
I20260929 20:50:36.169781 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40117
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42091
--webserver_port=34913
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:50:36 dist-test-slave-blv7 krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:50:36.184191 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:36.167116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34475 (local address 127.0.97.62:42091)
0929 20:50:36.167359 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:36.167364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:36.168546 (+ 1182us) server_negotiation.cc:407] Connection header received
0929 20:50:36.170469 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:36.170475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:36.170565 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:36.170677 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:50:36.172212 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.173120 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.174749 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.175369 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.175496 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:36.178805 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:50:36.178810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:50:36.178814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:50:36.178854 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:50:36.181206 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:36.181876 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:36.181879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:36.181882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:36.182010 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:36.182422 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:36.182426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:36.182428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:36.182713 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:50:36.182852 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:36.183577 (+ 725us) server_negotiation.cc:299] Negotiation successful
0929 20:50:36.183804 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260929 20:50:36.185267 16759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:36.167642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.2:34475)
0929 20:50:36.168380 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:36.168417 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:50:36.170180 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:50:36.170865 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:50:36.170873 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:50:36.171477 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:50:36.172043 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:36.172061 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.173293 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.173297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:36.174604 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:36.174616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.175581 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.175586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:36.175788 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:36.176421 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:50:36.176443 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:50:36.178579 (+ 2136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:50:36.181342 (+ 2763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:50:36.181351 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:50:36.181368 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:50:36.181761 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:50:36.182109 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:50:36.182113 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:50:36.182117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:50:36.182311 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:50:36.182829 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:50:36.182837 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:50:36.183242 (+ 405us) client_negotiation.cc:769] Sending connection context
0929 20:50:36.183502 (+ 260us) client_negotiation.cc:240] Negotiation successful
0929 20:50:36.184033 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":293,"threads_started":1}
I20260929 20:50:36.186724 16757 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42091
I20260929 20:50:36.187067 16757 heartbeater.cc:464] Registering TS with master...
I20260929 20:50:36.187795 16757 heartbeater.cc:511] Master 127.0.97.62:42091 requested a full tablet report, sending...
I20260929 20:50:36.189262 16440 ts_manager.cc:194] Registered new tserver with Master: 4f5d8a998dcb4ed1a8e536e5c40b5d63 (127.0.97.2:39055)
I20260929 20:50:36.190423 16440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34475
W20260929 20:50:36.490343 16760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:50:36.490633 16760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:50:36.490725 16760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:50:36.498072 16760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:50:36.498284 16760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:50:36.498446 16760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:50:36.508672 16760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34913
--tserver_master_addrs=127.0.97.62:42091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:50:36.510011 16760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:50:36.511363 16760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:50:36.525498 16766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:36.525514 16765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:36.526139 16768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:50:36.526465 16760 server_base.cc:1034] running on GCE node
I20260929 20:50:36.527076 16760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:50:36.528033 16760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:50:36.529312 16760 hybrid_clock.cc:648] HybridClock initialized: now 1790715036529248 us; error 52 us; skew 500 ppm
I20260929 20:50:36.529655 16760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:50:36 dist-test-slave-blv7 krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:50:36.532653 16760 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:50:36.534001 16760 webserver.cc:460] Webserver started at http://127.0.97.3:34913/ using document root <none> and password file <none>
I20260929 20:50:36.534657 16760 fs_manager.cc:359] Metadata directory not provided
I20260929 20:50:36.534783 16760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:50:36.540020 16760 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:50:36.542578 16775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:36.543890 16760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:50:36.544034 16760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "77d3674ed2f74c4cab409a034357df57"
format_stamp: "Formatted at 2026-09-29 20:50:13 on dist-test-slave-blv7"
server_key: "773e86a27cdbc1b758e2b9087a4bb884"
server_key_iv: "c3c30176bfb36b0de5002503dbf744fe"
server_key_version: "encryptionkey@0"
I20260929 20:50:36.544981 16760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:50:36.571686 16760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:50:36.576166 16760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:50:36.576443 16760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:50:36.577469 16760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:50:36.578926 16760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:50:36.579094 16760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 20:50:36.579272 16760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:50:36.579353 16760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:36.616132 16760 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40117
I20260929 20:50:36.616257 16887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40117 every 8 connection(s)
I20260929 20:50:36.617646 16760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:50:36.624539 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 16760
Sep 29 20:50:36 dist-test-slave-blv7 krb5kdc[15856](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:50:36.639153 16627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:36.620421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36541 (local address 127.0.97.62:42091)
0929 20:50:36.620601 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:36.620605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:36.621876 (+ 1271us) server_negotiation.cc:407] Connection header received
0929 20:50:36.623812 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:36.623818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:36.623911 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:36.623999 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:50:36.625772 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.626964 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.628681 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.629116 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.629232 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:36.632626 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:50:36.632633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:50:36.632638 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:50:36.632677 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:50:36.635810 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:36.636682 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:36.636687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:36.636691 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:36.636825 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:36.637321 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:36.637324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:36.637326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:36.637632 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:50:36.637767 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:36.638613 (+ 846us) server_negotiation.cc:299] Negotiation successful
0929 20:50:36.638802 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:50:36.640365 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:36.620831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.3:36541)
0929 20:50:36.621669 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:36.621708 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:50:36.623534 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:50:36.624281 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:50:36.624295 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:50:36.624882 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:50:36.625557 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:36.625584 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.627226 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.627231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:36.628548 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:36.628560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.629354 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.629363 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:36.629570 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:36.630186 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:50:36.630210 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:50:36.632434 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:50:36.635966 (+ 3532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:50:36.635975 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:50:36.635992 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:50:36.636570 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:50:36.636963 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:50:36.636968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:50:36.636972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:50:36.637221 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:50:36.637778 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:50:36.637786 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:50:36.638237 (+ 451us) client_negotiation.cc:769] Sending connection context
0929 20:50:36.638471 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 20:50:36.638951 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":334,"threads_started":1}
I20260929 20:50:36.640944 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:36.625428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57950 (local address 127.0.97.62:42091)
0929 20:50:36.627378 (+ 1950us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:36.627382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:36.627397 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:50:36.627435 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:36.627438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:36.627497 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:36.627587 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:50:36.629182 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.629940 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.631374 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.631814 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.631939 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:36.635003 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:50:36.635008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:50:36.635011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:50:36.635084 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:50:36.638031 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:36.638701 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:36.638707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:36.638710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:36.638857 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:36.639494 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:36.639500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:36.639503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:36.639819 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:50:36.640056 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:36.640307 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 20:50:36.640519 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1859,"thread_start_us":1730,"threads_started":1}
I20260929 20:50:36.641997 16888 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42091
I20260929 20:50:36.642318 16888 heartbeater.cc:464] Registering TS with master...
I20260929 20:50:36.643056 16888 heartbeater.cc:511] Master 127.0.97.62:42091 requested a full tablet report, sending...
I20260929 20:50:36.644387 16439 ts_manager.cc:194] Registered new tserver with Master: 77d3674ed2f74c4cab409a034357df57 (127.0.97.3:40117)
I20260929 20:50:36.645201 16439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36541
I20260929 20:50:36.655695 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:50:36.662187 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:36.656598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57956 (local address 127.0.97.62:42091)
0929 20:50:36.656796 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:36.656801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:36.657208 (+ 407us) server_negotiation.cc:407] Connection header received
0929 20:50:36.657467 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:36.657470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:36.657605 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:36.657729 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:50:36.658158 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.658893 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.660490 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.660903 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.660993 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:36.661477 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:50:36.661669 (+ 192us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 20:50:36.661951 (+ 282us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57956: token expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 20:50:36.662374 16892 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57956: token expired
W20260929 20:50:36.662846 16361 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:42091)
I20260929 20:50:36.674739 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:36.663342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57964 (local address 127.0.97.62:42091)
0929 20:50:36.663519 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:36.663522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:36.663537 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:50:36.663614 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:36.663617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:36.663670 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:36.663739 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:50:36.665046 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.665692 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.667101 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.667476 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.667558 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:36.669936 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:50:36.669941 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:50:36.669944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:50:36.669973 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:50:36.672405 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:36.672983 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:36.672987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:36.672990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:36.673106 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:36.673486 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:36.673488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:36.673491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:36.673790 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:50:36.673967 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:36.674268 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 20:50:36.674443 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 20:50:36.678263 16439 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:50:36.685650 16439 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:50:36.718923 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:36.698007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54916 (local address 127.0.97.3:40117)
0929 20:50:36.706885 (+ 8878us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:36.706897 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:36.708336 (+ 1439us) server_negotiation.cc:407] Connection header received
0929 20:50:36.708466 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:36.708475 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:36.708790 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:36.708990 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:50:36.711280 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.712792 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.716307 (+ 3515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.717687 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.717825 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:36.718060 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:36.718435 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 20:50:36.718609 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8596,"thread_start_us":8443,"threads_started":1}
I20260929 20:50:36.720427 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:36.698488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46078 (local address 127.0.97.1:33593)
0929 20:50:36.710159 (+ 11671us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:36.710169 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:36.710968 (+ 799us) server_negotiation.cc:407] Connection header received
0929 20:50:36.711316 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:36.711324 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:36.711699 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:36.711863 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:50:36.712723 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.714266 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.717870 (+ 3604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.719197 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.719294 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:36.719513 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:36.719955 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 20:50:36.720117 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11490,"thread_start_us":11679,"threads_started":1}
I20260929 20:50:36.720522 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:36.697961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40117 (local address 127.0.0.1:54916)
0929 20:50:36.707787 (+ 9826us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:36.707849 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 20:50:36.708079 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:50:36.710456 (+ 2377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:50:36.710464 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:50:36.710508 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:50:36.710992 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:36.711009 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.713541 (+ 2532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.713546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:36.716093 (+ 2547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:36.716108 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.717930 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.717938 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:36.718163 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:36.718221 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 20:50:36.718353 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 20:50:36.718574 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9529,"mutex_wait_us":49,"thread_start_us":18865,"threads_started":1}
I20260929 20:50:36.722770 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:36.698036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33593 (local address 127.0.0.1:46078)
0929 20:50:36.710842 (+ 12806us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:36.710863 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:50:36.711097 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:50:36.712142 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:50:36.712147 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:50:36.712171 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:50:36.712533 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:36.712545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.714452 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.714457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:36.717690 (+ 3233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:36.717704 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.719664 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.719668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:36.719795 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:36.719815 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 20:50:36.719890 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 20:50:36.720014 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12667,"mutex_wait_us":23,"thread_start_us":14885,"threads_started":1}
I20260929 20:50:36.724730 16823 tablet_service.cc:1467] Processing CreateTablet for tablet fd4de2f0550d498c8356f2bf9768ef59 (DEFAULT_TABLE table=test-table [id=4b22b9970a1849ac90337c2ec5bbb567]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:50:36.725495 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:36.699162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52680 (local address 127.0.97.2:39055)
0929 20:50:36.708214 (+ 9052us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:36.708235 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:36.716474 (+ 8239us) server_negotiation.cc:407] Connection header received
0929 20:50:36.717557 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:36.717565 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:36.717794 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:36.718009 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:50:36.718922 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.720461 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.722610 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.724121 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.724256 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:36.724453 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:36.725014 (+ 561us) server_negotiation.cc:299] Negotiation successful
0929 20:50:36.725185 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8290,"thread_start_us":15866,"threads_started":1}
I20260929 20:50:36.726011 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:36.698463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39055 (local address 127.0.0.1:52680)
0929 20:50:36.716361 (+ 17898us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:36.716380 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:50:36.717356 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:50:36.718273 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:50:36.718278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:50:36.718305 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:50:36.718708 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:36.718723 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.720947 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.720951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:36.722448 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:36.722461 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:36.724617 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:36.724624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:36.724793 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:36.724824 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:50:36.725548 (+ 724us) client_negotiation.cc:240] Negotiation successful
0929 20:50:36.725690 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17760,"thread_start_us":16805,"threads_started":1}
I20260929 20:50:36.726261 16823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd4de2f0550d498c8356f2bf9768ef59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:50:36.726540 16560 tablet_service.cc:1467] Processing CreateTablet for tablet fd4de2f0550d498c8356f2bf9768ef59 (DEFAULT_TABLE table=test-table [id=4b22b9970a1849ac90337c2ec5bbb567]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:50:36.727660 16560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd4de2f0550d498c8356f2bf9768ef59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:50:36.729688 16692 tablet_service.cc:1467] Processing CreateTablet for tablet fd4de2f0550d498c8356f2bf9768ef59 (DEFAULT_TABLE table=test-table [id=4b22b9970a1849ac90337c2ec5bbb567]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:50:36.731125 16692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd4de2f0550d498c8356f2bf9768ef59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:50:36.741508 16904 tablet_bootstrap.cc:492] T fd4de2f0550d498c8356f2bf9768ef59 P 275bc00f37b34367a4487c1ca7be4503: Bootstrap starting.
I20260929 20:50:36.744891 16904 tablet_bootstrap.cc:654] T fd4de2f0550d498c8356f2bf9768ef59 P 275bc00f37b34367a4487c1ca7be4503: Neither blocks nor log segments found. Creating new log.
I20260929 20:50:36.745790 16905 tablet_bootstrap.cc:492] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57: Bootstrap starting.
I20260929 20:50:36.746207 16904 log.cc:824] T fd4de2f0550d498c8356f2bf9768ef59 P 275bc00f37b34367a4487c1ca7be4503: Log is configured to *not* fsync() on all Append() calls
I20260929 20:50:36.746605 16625 heartbeater.cc:503] Master 127.0.97.62:42091 was elected leader, sending a full tablet report...
I20260929 20:50:36.749519 16905 tablet_bootstrap.cc:654] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57: Neither blocks nor log segments found. Creating new log.
I20260929 20:50:36.750975 16905 log.cc:824] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57: Log is configured to *not* fsync() on all Append() calls
I20260929 20:50:36.752925 16906 tablet_bootstrap.cc:492] T fd4de2f0550d498c8356f2bf9768ef59 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: Bootstrap starting.
I20260929 20:50:36.754755 16905 tablet_bootstrap.cc:492] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57: No bootstrap required, opened a new log
I20260929 20:50:36.754748 16904 tablet_bootstrap.cc:492] T fd4de2f0550d498c8356f2bf9768ef59 P 275bc00f37b34367a4487c1ca7be4503: No bootstrap required, opened a new log
I20260929 20:50:36.755060 16905 ts_tablet_manager.cc:1397] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 20:50:36.755074 16904 ts_tablet_manager.cc:1397] T fd4de2f0550d498c8356f2bf9768ef59 P 275bc00f37b34367a4487c1ca7be4503: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20260929 20:50:36.756414 16906 tablet_bootstrap.cc:654] T fd4de2f0550d498c8356f2bf9768ef59 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: Neither blocks nor log segments found. Creating new log.
I20260929 20:50:36.757592 16906 log.cc:824] T fd4de2f0550d498c8356f2bf9768ef59 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: Log is configured to *not* fsync() on all Append() calls
I20260929 20:50:36.760149 16906 tablet_bootstrap.cc:492] T fd4de2f0550d498c8356f2bf9768ef59 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: No bootstrap required, opened a new log
I20260929 20:50:36.760397 16906 ts_tablet_manager.cc:1397] T fd4de2f0550d498c8356f2bf9768ef59 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 20:50:36.765760 16904 raft_consensus.cc:348] T fd4de2f0550d498c8356f2bf9768ef59 P 275bc00f37b34367a4487c1ca7be4503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d3674ed2f74c4cab409a034357df57" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40117 } } peers { permanent_uuid: "275bc00f37b34367a4487c1ca7be4503" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39055 } }
I20260929 20:50:36.765838 16905 raft_consensus.cc:348] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d3674ed2f74c4cab409a034357df57" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40117 } } peers { permanent_uuid: "275bc00f37b34367a4487c1ca7be4503" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39055 } }
I20260929 20:50:36.766269 16904 raft_consensus.cc:374] T fd4de2f0550d498c8356f2bf9768ef59 P 275bc00f37b34367a4487c1ca7be4503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:50:36.766264 16905 raft_consensus.cc:374] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:50:36.766369 16905 raft_consensus.cc:729] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77d3674ed2f74c4cab409a034357df57, State: Initialized, Role: FOLLOWER
I20260929 20:50:36.766398 16904 raft_consensus.cc:729] T fd4de2f0550d498c8356f2bf9768ef59 P 275bc00f37b34367a4487c1ca7be4503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275bc00f37b34367a4487c1ca7be4503, State: Initialized, Role: FOLLOWER
I20260929 20:50:36.767104 16904 consensus_queue.cc:260] T fd4de2f0550d498c8356f2bf9768ef59 P 275bc00f37b34367a4487c1ca7be4503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d3674ed2f74c4cab409a034357df57" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40117 } } peers { permanent_uuid: "275bc00f37b34367a4487c1ca7be4503" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39055 } }
I20260929 20:50:36.767565 16905 consensus_queue.cc:260] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d3674ed2f74c4cab409a034357df57" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40117 } } peers { permanent_uuid: "275bc00f37b34367a4487c1ca7be4503" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39055 } }
I20260929 20:50:36.767588 16906 raft_consensus.cc:348] T fd4de2f0550d498c8356f2bf9768ef59 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d3674ed2f74c4cab409a034357df57" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40117 } } peers { permanent_uuid: "275bc00f37b34367a4487c1ca7be4503" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39055 } }
I20260929 20:50:36.768122 16906 raft_consensus.cc:374] T fd4de2f0550d498c8356f2bf9768ef59 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:50:36.768266 16906 raft_consensus.cc:729] T fd4de2f0550d498c8356f2bf9768ef59 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5d8a998dcb4ed1a8e536e5c40b5d63, State: Initialized, Role: FOLLOWER
I20260929 20:50:36.769003 16906 consensus_queue.cc:260] T fd4de2f0550d498c8356f2bf9768ef59 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d3674ed2f74c4cab409a034357df57" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40117 } } peers { permanent_uuid: "275bc00f37b34367a4487c1ca7be4503" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39055 } }
I20260929 20:50:36.769527 16904 ts_tablet_manager.cc:1428] T fd4de2f0550d498c8356f2bf9768ef59 P 275bc00f37b34367a4487c1ca7be4503: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260929 20:50:36.770512 16888 heartbeater.cc:503] Master 127.0.97.62:42091 was elected leader, sending a full tablet report...
I20260929 20:50:36.771178 16757 heartbeater.cc:503] Master 127.0.97.62:42091 was elected leader, sending a full tablet report...
I20260929 20:50:36.771747 16905 ts_tablet_manager.cc:1428] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57: Time spent starting tablet: real 0.016s user 0.012s sys 0.002s
I20260929 20:50:36.772877 16906 ts_tablet_manager.cc:1428] T fd4de2f0550d498c8356f2bf9768ef59 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20260929 20:50:36.869204 16889 tablet.cc:2369] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:50:36.916663 16758 tablet.cc:2369] T fd4de2f0550d498c8356f2bf9768ef59 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:50:36.961866 16626 tablet.cc:2369] T fd4de2f0550d498c8356f2bf9768ef59 P 275bc00f37b34367a4487c1ca7be4503: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:50:37.056540 16911 raft_consensus.cc:482] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:50:37.056857 16911 raft_consensus.cc:504] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d3674ed2f74c4cab409a034357df57" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40117 } } peers { permanent_uuid: "275bc00f37b34367a4487c1ca7be4503" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39055 } }
I20260929 20:50:37.058600 16911 leader_election.cc:288] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 275bc00f37b34367a4487c1ca7be4503 (127.0.97.1:33593), 4f5d8a998dcb4ed1a8e536e5c40b5d63 (127.0.97.2:39055)
I20260929 20:50:37.065267 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:37.059105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43151 (local address 127.0.97.2:39055)
0929 20:50:37.059337 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:37.059343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:37.059367 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:50:37.059517 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:37.059522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:37.059599 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:37.059681 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:50:37.060133 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:37.061102 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:37.063362 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:37.064476 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:37.064619 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:37.064716 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:37.064939 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 20:50:37.065066 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:50:37.065330 16890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:37.058924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39055 (local address 127.0.97.3:43151)
0929 20:50:37.059207 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:37.059226 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:50:37.059352 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:50:37.059733 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:50:37.059738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:50:37.059825 (+ 87us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:50:37.059992 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:37.060004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:37.061227 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:37.061231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:37.063131 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:37.063146 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:37.064636 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:37.064640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:37.064770 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:37.064794 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:50:37.064887 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 20:50:37.065016 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"spinlock_wait_cycles":24960}
I20260929 20:50:37.065716 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:37.058993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33593 (local address 127.0.97.3:46255)
0929 20:50:37.059639 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:37.059651 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 20:50:37.059763 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:50:37.060025 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:50:37.060030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:50:37.060051 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:50:37.060305 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:37.060312 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:37.061619 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:37.061624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:37.063501 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:50:37.063511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:50:37.065037 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:50:37.065042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:50:37.065205 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:37.065239 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:50:37.065340 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 20:50:37.065453 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":380,"threads_started":1}
I20260929 20:50:37.065724 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:37.059071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46255 (local address 127.0.97.1:33593)
0929 20:50:37.059241 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:37.059246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:37.059753 (+ 507us) server_negotiation.cc:407] Connection header received
0929 20:50:37.059861 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:37.059865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:37.059922 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:37.059988 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:50:37.060482 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:37.061488 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:37.063655 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:37.064842 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:37.064966 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:37.065058 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:37.065377 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 20:50:37.065506 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:50:37.066696 16712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4de2f0550d498c8356f2bf9768ef59" candidate_uuid: "77d3674ed2f74c4cab409a034357df57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" is_pre_election: true
I20260929 20:50:37.066823 16580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4de2f0550d498c8356f2bf9768ef59" candidate_uuid: "77d3674ed2f74c4cab409a034357df57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275bc00f37b34367a4487c1ca7be4503" is_pre_election: true
I20260929 20:50:37.067291 16712 raft_consensus.cc:2454] T fd4de2f0550d498c8356f2bf9768ef59 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77d3674ed2f74c4cab409a034357df57 in term 0.
I20260929 20:50:37.067346 16580 raft_consensus.cc:2454] T fd4de2f0550d498c8356f2bf9768ef59 P 275bc00f37b34367a4487c1ca7be4503 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77d3674ed2f74c4cab409a034357df57 in term 0.
I20260929 20:50:37.068010 16779 leader_election.cc:302] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 275bc00f37b34367a4487c1ca7be4503, 77d3674ed2f74c4cab409a034357df57; no voters:
I20260929 20:50:37.068377 16911 raft_consensus.cc:2781] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:50:37.068511 16911 raft_consensus.cc:482] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:50:37.068596 16911 raft_consensus.cc:3037] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:50:37.070670 16911 raft_consensus.cc:504] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d3674ed2f74c4cab409a034357df57" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40117 } } peers { permanent_uuid: "275bc00f37b34367a4487c1ca7be4503" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39055 } }
I20260929 20:50:37.071264 16911 leader_election.cc:288] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57 [CANDIDATE]: Term 1 election: Requested vote from peers 275bc00f37b34367a4487c1ca7be4503 (127.0.97.1:33593), 4f5d8a998dcb4ed1a8e536e5c40b5d63 (127.0.97.2:39055)
I20260929 20:50:37.071681 16580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4de2f0550d498c8356f2bf9768ef59" candidate_uuid: "77d3674ed2f74c4cab409a034357df57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "275bc00f37b34367a4487c1ca7be4503"
I20260929 20:50:37.071767 16712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd4de2f0550d498c8356f2bf9768ef59" candidate_uuid: "77d3674ed2f74c4cab409a034357df57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63"
I20260929 20:50:37.071892 16580 raft_consensus.cc:3037] T fd4de2f0550d498c8356f2bf9768ef59 P 275bc00f37b34367a4487c1ca7be4503 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:50:37.071954 16712 raft_consensus.cc:3037] T fd4de2f0550d498c8356f2bf9768ef59 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:50:37.074915 16712 raft_consensus.cc:2454] T fd4de2f0550d498c8356f2bf9768ef59 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77d3674ed2f74c4cab409a034357df57 in term 1.
I20260929 20:50:37.074915 16580 raft_consensus.cc:2454] T fd4de2f0550d498c8356f2bf9768ef59 P 275bc00f37b34367a4487c1ca7be4503 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77d3674ed2f74c4cab409a034357df57 in term 1.
I20260929 20:50:37.075332 16777 leader_election.cc:302] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f5d8a998dcb4ed1a8e536e5c40b5d63, 77d3674ed2f74c4cab409a034357df57; no voters:
I20260929 20:50:37.075613 16911 raft_consensus.cc:2781] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:50:37.076076 16911 raft_consensus.cc:686] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57 [term 1 LEADER]: Becoming Leader. State: Replica: 77d3674ed2f74c4cab409a034357df57, State: Running, Role: LEADER
I20260929 20:50:37.076591 16911 consensus_queue.cc:237] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d3674ed2f74c4cab409a034357df57" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40117 } } peers { permanent_uuid: "275bc00f37b34367a4487c1ca7be4503" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } } peers { permanent_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39055 } }
I20260929 20:50:37.080798 16438 catalog_manager.cc:5495] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77d3674ed2f74c4cab409a034357df57 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "77d3674ed2f74c4cab409a034357df57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77d3674ed2f74c4cab409a034357df57" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "275bc00f37b34367a4487c1ca7be4503" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39055 } health_report { overall_health: UNKNOWN } } }
I20260929 20:50:37.182503 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:37.177313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.0.97.3:40117)
0929 20:50:37.177509 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:37.177513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:37.177746 (+ 233us) server_negotiation.cc:407] Connection header received
0929 20:50:37.177910 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:37.177914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:37.177996 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:37.178122 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:50:37.178512 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:37.179342 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:37.180685 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:37.181076 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:37.181167 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:37.181587 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:50:37.182092 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:50:37.182198 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:37.182239 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 20:50:37.182310 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:50:37.190527 16712 raft_consensus.cc:1261] T fd4de2f0550d498c8356f2bf9768ef59 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [term 1 FOLLOWER]: Refusing update from remote peer 77d3674ed2f74c4cab409a034357df57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:50:37.190572 16580 raft_consensus.cc:1261] T fd4de2f0550d498c8356f2bf9768ef59 P 275bc00f37b34367a4487c1ca7be4503 [term 1 FOLLOWER]: Refusing update from remote peer 77d3674ed2f74c4cab409a034357df57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:50:37.191480 16914 consensus_queue.cc:1035] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "275bc00f37b34367a4487c1ca7be4503" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:50:37.191819 16911 consensus_queue.cc:1035] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:50:37.201112 16918 mvcc.cc:205] Tried to move back new op lower bound from 7334768792322789376 to 7334768791869526016. Current Snapshot: MvccSnapshot[applied={T|T < 7334768792322789376}]
I20260929 20:50:37.203119 16920 mvcc.cc:205] Tried to move back new op lower bound from 7334768792322789376 to 7334768791869526016. Current Snapshot: MvccSnapshot[applied={T|T < 7334768792322789376}]
I20260929 20:50:37.217928 16440 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:50:37.218179 16440 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:50:37.221602 16440 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379: Sending DeleteTablet for 3 replicas of tablet fd4de2f0550d498c8356f2bf9768ef59
I20260929 20:50:37.222664 16823 tablet_service.cc:1514] Processing DeleteTablet for tablet fd4de2f0550d498c8356f2bf9768ef59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:50:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54916
I20260929 20:50:37.222815 16560 tablet_service.cc:1514] Processing DeleteTablet for tablet fd4de2f0550d498c8356f2bf9768ef59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:50:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46078
I20260929 20:50:37.222896 16692 tablet_service.cc:1514] Processing DeleteTablet for tablet fd4de2f0550d498c8356f2bf9768ef59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:50:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52680
I20260929 20:50:37.223057 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 16477
I20260929 20:50:37.236603 16930 tablet_replica.cc:330] T fd4de2f0550d498c8356f2bf9768ef59 P 275bc00f37b34367a4487c1ca7be4503: stopping tablet replica
I20260929 20:50:37.240298 16930 raft_consensus.cc:2229] T fd4de2f0550d498c8356f2bf9768ef59 P 275bc00f37b34367a4487c1ca7be4503 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:50:37.240954 16930 raft_consensus.cc:2258] T fd4de2f0550d498c8356f2bf9768ef59 P 275bc00f37b34367a4487c1ca7be4503 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:50:37.241518 16931 tablet_replica.cc:330] T fd4de2f0550d498c8356f2bf9768ef59 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: stopping tablet replica
I20260929 20:50:37.242269 16931 raft_consensus.cc:2229] T fd4de2f0550d498c8356f2bf9768ef59 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:50:37.243463 16929 tablet_replica.cc:330] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57: stopping tablet replica
I20260929 20:50:37.243841 16929 raft_consensus.cc:2229] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:50:37.244458 16929 raft_consensus.cc:2258] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:50:37.244889 16930 ts_tablet_manager.cc:1905] T fd4de2f0550d498c8356f2bf9768ef59 P 275bc00f37b34367a4487c1ca7be4503: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:50:37.246285 16929 ts_tablet_manager.cc:1905] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:50:37.252056 16931 raft_consensus.cc:2258] T fd4de2f0550d498c8356f2bf9768ef59 P 4f5d8a998dcb4ed1a8e536e5c40b5d63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:50:37.252679 16929 ts_tablet_manager.cc:1918] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:50:37.252871 16930 ts_tablet_manager.cc:1918] T fd4de2f0550d498c8356f2bf9768ef59 P 275bc00f37b34367a4487c1ca7be4503: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:50:37.252913 16929 log.cc:1196] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/fd4de2f0550d498c8356f2bf9768ef59
I20260929 20:50:37.253064 16930 log.cc:1196] T fd4de2f0550d498c8356f2bf9768ef59 P 275bc00f37b34367a4487c1ca7be4503: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/fd4de2f0550d498c8356f2bf9768ef59
I20260929 20:50:37.253566 16929 ts_tablet_manager.cc:1939] T fd4de2f0550d498c8356f2bf9768ef59 P 77d3674ed2f74c4cab409a034357df57: Deleting consensus metadata
I20260929 20:50:37.253621 16930 ts_tablet_manager.cc:1939] T fd4de2f0550d498c8356f2bf9768ef59 P 275bc00f37b34367a4487c1ca7be4503: Deleting consensus metadata
I20260929 20:50:37.254236 16931 ts_tablet_manager.cc:1905] T fd4de2f0550d498c8356f2bf9768ef59 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:50:37.255410 16427 catalog_manager.cc:4841] TS 275bc00f37b34367a4487c1ca7be4503 (127.0.97.1:33593): tablet fd4de2f0550d498c8356f2bf9768ef59 (table test-table [id=4b22b9970a1849ac90337c2ec5bbb567]) successfully deleted
I20260929 20:50:37.255434 16426 catalog_manager.cc:4841] TS 77d3674ed2f74c4cab409a034357df57 (127.0.97.3:40117): tablet fd4de2f0550d498c8356f2bf9768ef59 (table test-table [id=4b22b9970a1849ac90337c2ec5bbb567]) successfully deleted
I20260929 20:50:37.260206 16931 ts_tablet_manager.cc:1918] T fd4de2f0550d498c8356f2bf9768ef59 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:50:37.260371 16931 log.cc:1196] T fd4de2f0550d498c8356f2bf9768ef59 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/fd4de2f0550d498c8356f2bf9768ef59
I20260929 20:50:37.260792 16931 ts_tablet_manager.cc:1939] T fd4de2f0550d498c8356f2bf9768ef59 P 4f5d8a998dcb4ed1a8e536e5c40b5d63: Deleting consensus metadata
I20260929 20:50:37.262367 16425 catalog_manager.cc:4841] TS 4f5d8a998dcb4ed1a8e536e5c40b5d63 (127.0.97.2:39055): tablet fd4de2f0550d498c8356f2bf9768ef59 (table test-table [id=4b22b9970a1849ac90337c2ec5bbb567]) successfully deleted
I20260929 20:50:37.265288 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:37.223896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46084 (local address 127.0.97.1:33593)
0929 20:50:37.224143 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:37.224151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:37.232948 (+ 8797us) server_negotiation.cc:407] Connection header received
0929 20:50:37.233476 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:37.233483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:37.233600 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:37.233682 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:50:37.236838 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:37.237959 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:37.242591 (+ 4632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:37.243292 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:37.243391 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:37.258262 (+ 14871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:50:37.258278 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:50:37.258296 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:50:37.258343 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:50:37.262219 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:37.263065 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:37.263076 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:37.263080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:37.263249 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:37.263604 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:37.263609 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:37.263612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:37.264094 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:50:37.264388 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:37.264675 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 20:50:37.264916 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 20:50:37.265941 16620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:50:37.354460 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 16477
I20260929 20:50:37.375216 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 16629
I20260929 20:50:37.390785 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:37.375952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52684 (local address 127.0.97.2:39055)
0929 20:50:37.376126 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:37.376130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:37.376147 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:50:37.376316 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:37.376321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:37.376413 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:37.376494 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:50:37.378241 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:37.378955 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:37.380951 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:37.381357 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:37.381451 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:37.385204 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:50:37.385220 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:50:37.385236 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:50:37.385282 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:50:37.387994 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:37.388828 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:37.388837 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:37.388839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:37.388966 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:37.389363 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:37.389366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:37.389368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:37.389720 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:50:37.389927 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:37.390321 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 20:50:37.390508 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:50:37.391513 16752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:50:37.479255 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 16629
I20260929 20:50:37.499173 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 16760
I20260929 20:50:37.513906 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:37.500073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54942 (local address 127.0.97.3:40117)
0929 20:50:37.500279 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:50:37.500286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:50:37.500304 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:50:37.500363 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:50:37.500367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:50:37.500434 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:50:37.500528 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:50:37.501815 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:37.502637 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:37.504158 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:50:37.504619 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:50:37.504741 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:50:37.507794 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:50:37.507807 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:50:37.507823 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:50:37.507870 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:50:37.511122 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:37.511873 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:37.511881 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:37.511883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:37.512043 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:50:37.512404 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:50:37.512409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:50:37.512411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:50:37.512767 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:50:37.513042 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 20:50:37.513316 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 20:50:37.513535 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:50:37.514740 16883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:50:37.634043 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 16760
I20260929 20:50:37.649333 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 16407
I20260929 20:50:37.650452 16470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:50:37.761235 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 16407
I20260929 20:50:57.778672 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42091
--webserver_interface=127.0.97.62
--webserver_port=38015
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:50:58.096653 16937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:50:58.096956 16937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:50:58.097069 16937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:50:58.104439 16937 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 20:50:58.104542 16937 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 20:50:58.104591 16937 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 20:50:58.104647 16937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:50:58.104707 16937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:50:58.104777 16937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:50:58.104827 16937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:50:58.104887 16937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:50:58.114605 16937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:50:58.115816 16937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:50:58.117082 16937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:50:58.128845 16943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:58.128880 16942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:58.129315 16945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:50:58.129773 16937 server_base.cc:1034] running on GCE node
I20260929 20:50:58.130462 16937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:50:58.131784 16937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:50:58.133008 16937 hybrid_clock.cc:648] HybridClock initialized: now 1790715058132915 us; error 71 us; skew 500 ppm
I20260929 20:50:58.133287 16937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:50:58 dist-test-slave-blv7 krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:50:58.136482 16937 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:50:58.137835 16937 webserver.cc:460] Webserver started at http://127.0.97.62:38015/ using document root <none> and password file <none>
I20260929 20:50:58.138479 16937 fs_manager.cc:359] Metadata directory not provided
I20260929 20:50:58.138561 16937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:50:58.144161 16937 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:50:58.146852 16952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:58.148233 16937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:50:58.148403 16937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "cab36775e31448d180fd3daf0675b379"
format_stamp: "Formatted at 2026-09-29 20:50:12 on dist-test-slave-blv7"
server_key: "e8a34a1b6a61a012417edb04677f2da4"
server_key_iv: "0331bd94cabc8f407f23bba738328061"
server_key_version: "encryptionkey@0"
I20260929 20:50:58.149415 16937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:50:58.165252 16937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:50:58.169176 16937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:50:58.169435 16937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:50:58.186456 16937 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42091
I20260929 20:50:58.186504 17003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42091 every 8 connection(s)
I20260929 20:50:58.187995 16937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:50:58.190090 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 16937
I20260929 20:50:58.190900 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33593
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42091
--webserver_port=41535
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:50:58.195590 17004 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:50:58.198985 17004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379: Bootstrap starting.
I20260929 20:50:58.219954 17004 log.cc:824] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379: Log is configured to *not* fsync() on all Append() calls
I20260929 20:50:58.237413 17004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:50:58.237978 17004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379: Bootstrap complete.
I20260929 20:50:58.250196 17004 raft_consensus.cc:348] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab36775e31448d180fd3daf0675b379" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42091 } }
I20260929 20:50:58.251171 17004 raft_consensus.cc:729] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab36775e31448d180fd3daf0675b379, State: Initialized, Role: FOLLOWER
I20260929 20:50:58.251848 17004 consensus_queue.cc:260] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab36775e31448d180fd3daf0675b379" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42091 } }
I20260929 20:50:58.252085 17004 raft_consensus.cc:388] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:50:58.252214 17004 raft_consensus.cc:482] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:50:58.252381 17004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 2 FOLLOWER]: Advancing to term 3
I20260929 20:50:58.256527 17004 raft_consensus.cc:504] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab36775e31448d180fd3daf0675b379" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42091 } }
I20260929 20:50:58.257021 17004 leader_election.cc:302] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cab36775e31448d180fd3daf0675b379; no voters:
I20260929 20:50:58.257699 17004 leader_election.cc:288] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 20:50:58.257863 17008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 20:50:58.258493 17008 raft_consensus.cc:686] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [term 3 LEADER]: Becoming Leader. State: Replica: cab36775e31448d180fd3daf0675b379, State: Running, Role: LEADER
I20260929 20:50:58.259253 17008 consensus_queue.cc:237] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab36775e31448d180fd3daf0675b379" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42091 } }
I20260929 20:50:58.261402 17008 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cab36775e31448d180fd3daf0675b379. Latest consensus state: current_term: 3 leader_uuid: "cab36775e31448d180fd3daf0675b379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab36775e31448d180fd3daf0675b379" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42091 } } }
I20260929 20:50:58.261770 17008 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [sys.catalog]: This master's current role is: LEADER
I20260929 20:50:58.261891 17010 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cab36775e31448d180fd3daf0675b379" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab36775e31448d180fd3daf0675b379" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42091 } } }
I20260929 20:50:58.262252 17010 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [sys.catalog]: This master's current role is: LEADER
I20260929 20:50:58.262995 17004 sys_catalog.cc:564] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:50:58.268193 17012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:50:58.276850 17012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:50:58.278299 17012 catalog_manager.cc:1259] Loaded cluster ID: 2108cd2f26b5401bab680f8c3f0ae652
I20260929 20:50:58.278472 17012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:50:58.282675 17012 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:50:58.284350 17012 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379: Loaded TSK: 0
I20260929 20:50:58.284572 17012 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cab36775e31448d180fd3daf0675b379: Loaded TSK: 1
F20260929 20:50:58.285660 17012 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe4bef789ad google::LogMessage::Fail() at ??:0
@ 0x7fe4bef7b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe4bef784de google::LogMessage::Flush() at ??:0
@ 0x7fe4bef79a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe4c2cdc7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe4c2ccb73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe4d49a71c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe4d49a6783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe4d4a322a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fe4d4a320c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fe4d4a7c5e9 std::function<>::operator()() at ??:0
@ 0x7fe4d49aacdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fe4d49a8c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe4d4a30610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fe4d4a30439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe4c1bc4345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe4c1bc791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe4c1bc7799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe4c1ba2eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe4c4e7c6db start_thread at ??:0
@ 0x7fe4bde1261f clone at ??:0
W20260929 20:50:58.565374 17006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:50:58.565642 17006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:50:58.565763 17006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:50:58.573158 17006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:50:58.573259 17006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:50:58.573431 17006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:50:58.582994 17006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41535
--tserver_master_addrs=127.0.97.62:42091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:50:58.584228 17006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:50:58.585501 17006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:50:58.599251 17054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:58.599334 17051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:58.599287 17052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:50:58.599992 17006 server_base.cc:1034] running on GCE node
I20260929 20:50:58.600696 17006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:50:58.601737 17006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:50:58.602977 17006 hybrid_clock.cc:648] HybridClock initialized: now 1790715058602917 us; error 48 us; skew 500 ppm
I20260929 20:50:58.603344 17006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:50:58 dist-test-slave-blv7 krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:50:58.606508 17006 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:50:58.608024 17006 webserver.cc:460] Webserver started at http://127.0.97.1:41535/ using document root <none> and password file <none>
I20260929 20:50:58.608705 17006 fs_manager.cc:359] Metadata directory not provided
I20260929 20:50:58.608837 17006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:50:58.614285 17006 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.003s
I20260929 20:50:58.617038 17061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:58.618381 17006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:50:58.618533 17006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "275bc00f37b34367a4487c1ca7be4503"
format_stamp: "Formatted at 2026-09-29 20:50:12 on dist-test-slave-blv7"
server_key: "88e62f108539b678fb39c74a5ba4f7cf"
server_key_iv: "3e03b02609822a64d29a6e5a33e23a84"
server_key_version: "encryptionkey@0"
I20260929 20:50:58.619532 17006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:50:58.647282 17006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:50:58.651695 17006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:50:58.651957 17006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:50:58.652936 17006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:50:58.654225 17006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:50:58.654316 17006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:58.654419 17006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:50:58.654487 17006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:58.684048 17006 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33593
I20260929 20:50:58.684116 17173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33593 every 8 connection(s)
I20260929 20:50:58.685433 17006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:50:58.689986 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:58.688236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.1:36741)
0929 20:50:58.688935 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:58.689133 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":250,"threads_started":1}
W20260929 20:50:58.690611 17174 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42091 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:42091: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
I20260929 20:50:58.691401 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:58.691105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.1:49133)
0929 20:50:58.691222 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:58.691302 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 20:50:58.692111 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:58.691839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.1:50673)
0929 20:50:58.691948 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:58.692014 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 20:50:58.692337 17174 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 20:50:58.692903 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 17006
I20260929 20:50:58.693468 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:39055
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42091
--webserver_port=36571
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:50:59.003340 17177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:50:59.003624 17177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:50:59.003772 17177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:50:59.010879 17177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:50:59.010999 17177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:50:59.011168 17177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:50:59.020586 17177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:39055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36571
--tserver_master_addrs=127.0.97.62:42091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:50:59.021781 17177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:50:59.023082 17177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:50:59.039451 17185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:59.039491 17183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:59.039510 17182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:50:59.040257 17177 server_base.cc:1034] running on GCE node
I20260929 20:50:59.040815 17177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:50:59.041693 17177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:50:59.042892 17177 hybrid_clock.cc:648] HybridClock initialized: now 1790715059042843 us; error 37 us; skew 500 ppm
I20260929 20:50:59.043213 17177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:50:59 dist-test-slave-blv7 krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:50:59.046248 17177 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:50:59.047703 17177 webserver.cc:460] Webserver started at http://127.0.97.2:36571/ using document root <none> and password file <none>
I20260929 20:50:59.048336 17177 fs_manager.cc:359] Metadata directory not provided
I20260929 20:50:59.048434 17177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:50:59.053540 17177 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 20:50:59.056169 17192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:59.057423 17177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:50:59.057567 17177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "4f5d8a998dcb4ed1a8e536e5c40b5d63"
format_stamp: "Formatted at 2026-09-29 20:50:13 on dist-test-slave-blv7"
server_key: "3345e601c291499b972316c59af06cfa"
server_key_iv: "bb8b0013e063c7f43f03a82fdf7fbe62"
server_key_version: "encryptionkey@0"
I20260929 20:50:59.058523 17177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:50:59.075904 17177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:50:59.080014 17177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:50:59.080267 17177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:50:59.081259 17177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:50:59.082559 17177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:50:59.082682 17177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:59.082789 17177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:50:59.082863 17177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:59.111881 17177 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39055
I20260929 20:50:59.111937 17304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39055 every 8 connection(s)
I20260929 20:50:59.113279 17177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:50:59.113837 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 17177
I20260929 20:50:59.114558 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40117
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42091
--webserver_port=34913
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:50:59.118191 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:59.116237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.2:35855)
0929 20:50:59.117046 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:59.117293 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":323,"threads_started":1}
W20260929 20:50:59.118795 17305 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42091 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:42091: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
I20260929 20:50:59.119602 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:59.119300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.2:53495)
0929 20:50:59.119431 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:59.119498 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 20:50:59.120375 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:59.120077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.2:47295)
0929 20:50:59.120193 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:59.120273 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 20:50:59.120595 17305 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:50:59.424979 17308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:50:59.425278 17308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:50:59.425419 17308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:50:59.432626 17308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:50:59.432749 17308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:50:59.432897 17308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:50:59.442585 17308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34913
--tserver_master_addrs=127.0.97.62:42091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:50:59.443859 17308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:50:59.445129 17308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:50:59.457957 17314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:59.457899 17316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:50:59.458189 17313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:50:59.459010 17308 server_base.cc:1034] running on GCE node
I20260929 20:50:59.459637 17308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:50:59.460597 17308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:50:59.461805 17308 hybrid_clock.cc:648] HybridClock initialized: now 1790715059461743 us; error 50 us; skew 500 ppm
I20260929 20:50:59.462105 17308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:50:59 dist-test-slave-blv7 krb5kdc[15856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:50:59.469939 17308 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:50:59.471397 17308 webserver.cc:460] Webserver started at http://127.0.97.3:34913/ using document root <none> and password file <none>
I20260929 20:50:59.472043 17308 fs_manager.cc:359] Metadata directory not provided
I20260929 20:50:59.472137 17308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:50:59.477252 17308 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 20:50:59.481348 17323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:59.482687 17308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:50:59.482834 17308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "77d3674ed2f74c4cab409a034357df57"
format_stamp: "Formatted at 2026-09-29 20:50:13 on dist-test-slave-blv7"
server_key: "773e86a27cdbc1b758e2b9087a4bb884"
server_key_iv: "c3c30176bfb36b0de5002503dbf744fe"
server_key_version: "encryptionkey@0"
I20260929 20:50:59.483856 17308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:50:59.530710 17308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:50:59.534511 17308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:50:59.534757 17308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:50:59.535789 17308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:50:59.537052 17308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:50:59.537134 17308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:59.537235 17308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:50:59.537302 17308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:50:59.566797 17308 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40117
I20260929 20:50:59.566864 17435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40117 every 8 connection(s)
I20260929 20:50:59.568282 17308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:50:59.572909 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:59.571137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.3:60829)
0929 20:50:59.571855 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:59.572056 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":270,"threads_started":1}
W20260929 20:50:59.573473 17436 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42091 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:42091: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
I20260929 20:50:59.574183 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:59.573906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.3:48529)
0929 20:50:59.574017 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:59.574084 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 20:50:59.574856 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:59.574579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.3:43427)
0929 20:50:59.574695 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:59.574761 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 20:50:59.575073 17436 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 20:50:59.576397 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 17308
I20260929 20:50:59.693612 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:50:59.692849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.1:47025)
0929 20:50:59.693282 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 20:50:59.693370 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":201,"threads_started":1}
I20260929 20:51:00.122025 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:00.121214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.2:45073)
0929 20:51:00.121663 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:00.121760 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":234,"threads_started":1}
I20260929 20:51:00.576181 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:00.575548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.3:45027)
0929 20:51:00.575919 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:00.575998 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":180,"threads_started":1}
I20260929 20:51:00.695225 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:00.694476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.1:49197)
0929 20:51:00.694921 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:00.695009 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":220,"threads_started":1}
I20260929 20:51:01.123653 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:01.122859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.2:54085)
0929 20:51:01.123348 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:01.123455 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":252,"threads_started":1}
I20260929 20:51:01.577696 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:01.577028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.3:53827)
0929 20:51:01.577465 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:01.577547 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":211,"threads_started":1}
I20260929 20:51:01.696775 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:01.696115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.1:60787)
0929 20:51:01.696529 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:01.696610 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":219,"threads_started":1}
I20260929 20:51:02.125285 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:02.124559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.2:47037)
0929 20:51:02.125032 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:02.125113 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":236,"threads_started":1}
I20260929 20:51:02.579373 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:02.578595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.3:47505)
0929 20:51:02.579098 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:02.579204 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":246,"threads_started":1}
I20260929 20:51:02.698446 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:02.697742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.1:37089)
0929 20:51:02.698162 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:02.698247 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":200,"threads_started":1}
I20260929 20:51:03.127012 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:03.126258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.2:40077)
0929 20:51:03.126736 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:03.126817 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":238,"threads_started":1}
I20260929 20:51:03.580910 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:03.580236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.3:39285)
0929 20:51:03.580674 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:03.580753 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":241,"threads_started":1}
I20260929 20:51:03.700238 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:03.699493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.1:58283)
0929 20:51:03.699982 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:03.700067 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":241,"threads_started":1}
I20260929 20:51:04.128751 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:04.127968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.2:52327)
0929 20:51:04.128474 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:04.128569 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":248,"threads_started":1}
I20260929 20:51:04.582782 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:04.581898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.3:34001)
0929 20:51:04.582463 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:04.582560 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":284,"threads_started":1}
I20260929 20:51:04.702020 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:04.701113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.1:58171)
0929 20:51:04.701655 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:04.701831 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":281,"threads_started":1}
I20260929 20:51:05.130388 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:05.129670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.2:55295)
0929 20:51:05.130131 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:05.130221 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":234,"threads_started":1}
I20260929 20:51:05.584506 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:05.583766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.3:34719)
0929 20:51:05.584233 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:05.584319 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":237,"threads_started":1}
I20260929 20:51:05.703878 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:05.702975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.1:43589)
0929 20:51:05.703534 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:05.703623 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":233,"threads_started":1}
I20260929 20:51:06.132112 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:06.131322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.2:56421)
0929 20:51:06.131793 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:06.131897 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":200,"threads_started":1}
I20260929 20:51:06.586432 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:06.585498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.3:56771)
0929 20:51:06.586091 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:06.586192 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":243,"threads_started":1}
I20260929 20:51:06.705607 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:06.704893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.1:37759)
0929 20:51:06.705346 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:06.705438 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":221,"threads_started":1}
I20260929 20:51:07.133697 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:07.133015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.2:45255)
0929 20:51:07.133461 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:07.133540 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":256,"threads_started":1}
I20260929 20:51:07.588198 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:07.587433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.3:44791)
0929 20:51:07.587924 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:07.588008 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":269,"threads_started":1}
I20260929 20:51:07.707388 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:07.706581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.1:60225)
0929 20:51:07.707138 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:07.707215 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":289,"threads_started":1}
I20260929 20:51:08.135440 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:08.134677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.2:56465)
0929 20:51:08.135154 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:08.135256 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":208,"threads_started":1}
I20260929 20:51:08.589821 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:08.589074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.3:59733)
0929 20:51:08.589549 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:08.589636 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":236,"threads_started":1}
I20260929 20:51:08.708926 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:08.708261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.1:54631)
0929 20:51:08.708691 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:08.708769 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":229,"threads_started":1}
I20260929 20:51:09.137157 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:09.136374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.2:58381)
0929 20:51:09.136915 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:09.136992 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":321,"threads_started":1}
I20260929 20:51:09.591441 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:09.590635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.3:36283)
0929 20:51:09.591127 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:09.591216 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":235,"threads_started":1}
I20260929 20:51:09.710525 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:09.709792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.1:57021)
0929 20:51:09.710259 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:09.710350 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":225,"threads_started":1}
I20260929 20:51:10.138810 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:10.138042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.2:38513)
0929 20:51:10.138517 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:10.138605 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":226,"threads_started":1}
I20260929 20:51:10.593004 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:10.592349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.3:53327)
0929 20:51:10.592753 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:10.592831 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":214,"threads_started":1}
I20260929 20:51:10.712109 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:10.711397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.1:35441)
0929 20:51:10.711851 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:10.711941 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":227,"threads_started":1}
I20260929 20:51:11.140512 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:11.139852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.2:51027)
0929 20:51:11.140270 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:11.140360 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":206,"threads_started":1}
I20260929 20:51:11.594631 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:11.593939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.3:50595)
0929 20:51:11.594374 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:11.594458 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":242,"threads_started":1}
I20260929 20:51:11.713747 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:11.712988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.1:33793)
0929 20:51:11.713435 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:11.713522 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":204,"threads_started":1}
I20260929 20:51:12.142306 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:12.141529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.2:50593)
0929 20:51:12.142027 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:12.142113 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":247,"threads_started":1}
I20260929 20:51:12.596254 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:12.595449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.3:53285)
0929 20:51:12.595968 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:12.596061 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":301,"threads_started":1}
I20260929 20:51:12.715305 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:12.714605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.1:57935)
0929 20:51:12.715005 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:12.715124 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":196,"threads_started":1}
I20260929 20:51:13.143980 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:13.143191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.2:44433)
0929 20:51:13.143715 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:13.143797 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":222,"threads_started":1}
I20260929 20:51:13.597963 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:13.597189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.3:58319)
0929 20:51:13.597688 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:13.597771 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":219,"threads_started":1}
I20260929 20:51:13.717073 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:13.716340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.1:56111)
0929 20:51:13.716800 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:13.716891 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":249,"threads_started":1}
I20260929 20:51:14.145946 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:14.145159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.2:52087)
0929 20:51:14.145664 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:14.145751 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":286,"threads_started":1}
I20260929 20:51:14.587877 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 17006
I20260929 20:51:14.599794 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:14.598959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42091 (local address 127.0.97.3:50391)
0929 20:51:14.599449 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:14.599586 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":242,"threads_started":1}
I20260929 20:51:14.605414 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:14.589074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53854 (local address 127.0.97.1:33593)
0929 20:51:14.589567 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:14.589581 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:14.589618 (+ 37us) server_negotiation.cc:407] Connection header received
0929 20:51:14.591176 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:14.591185 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:14.591609 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:14.591954 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:14.593365 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:14.594577 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:14.595865 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:14.596507 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:14.596667 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:14.599319 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:14.599329 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:14.599348 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:14.599394 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:14.602280 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:14.602967 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:14.602974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:14.602982 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:14.603198 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:14.603595 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:14.603598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:14.603600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:14.604120 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:14.604339 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:14.604793 (+ 454us) server_negotiation.cc:299] Negotiation successful
0929 20:51:14.605143 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":222,"threads_started":1}
I20260929 20:51:14.606712 17169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:14.691807 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 17006
I20260929 20:51:14.705392 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 17177
I20260929 20:51:14.722082 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:14.706430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35918 (local address 127.0.97.2:39055)
0929 20:51:14.707043 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:14.707062 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:14.707103 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:51:14.708606 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:14.708617 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:14.709044 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:14.709327 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:14.710633 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:14.711918 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:14.713207 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:14.713870 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:14.714017 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:14.716414 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:14.716421 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:14.716437 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:14.716477 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:14.719165 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:14.719854 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:14.719860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:14.719867 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:14.720067 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:14.720421 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:14.720424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:14.720426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:14.720896 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:14.721178 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:14.721479 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 20:51:14.721792 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":252,"threads_started":1}
I20260929 20:51:14.723366 17300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:14.805111 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 17177
I20260929 20:51:14.819034 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 17308
I20260929 20:51:14.836607 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:14.819924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42558 (local address 127.0.97.3:40117)
0929 20:51:14.820496 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:14.820512 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:14.820554 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:51:14.822129 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:14.822139 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:14.822577 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:14.822867 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:14.824295 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:14.825521 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:14.826857 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:14.827583 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:14.827720 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:14.830251 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:14.830258 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:14.830274 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:14.830309 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:14.833329 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:14.834100 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:14.834106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:14.834113 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:14.834273 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:14.834638 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:14.834641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:14.834643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:14.835219 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:14.835431 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:14.835929 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 20:51:14.836260 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":252,"threads_started":1}
I20260929 20:51:14.837996 17431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:14.920146 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 17308
2026-09-29T20:51:14Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64232 ms)
[----------] 30 tests from SecurityITest (288244 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:51:15 dist-test-slave-blv7 krb5kdc[17493](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:51:15 dist-test-slave-blv7 krb5kdc[17493](info): set up 2 sockets
Sep 29 20:51:15 dist-test-slave-blv7 krb5kdc[17493](info): commencing operation
krb5kdc: starting...
W20260929 20:51:15.926918 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:51:16 dist-test-slave-blv7 krb5kdc[17493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:51:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:51:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:51:16.150665 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46731
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:16.471678 17506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:16.471951 17506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:16.472048 17506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:16.479135 17506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:51:16.479233 17506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:16.479283 17506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:16.479337 17506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:51:16.479386 17506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:51:16.489079 17506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:16.490298 17506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:16.491616 17506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:16.503173 17514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:16.503285 17511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:16.503315 17512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:16.504274 17506 server_base.cc:1034] running on GCE node
I20260929 20:51:16.505120 17506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:16.506524 17506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:16.507732 17506 hybrid_clock.cc:648] HybridClock initialized: now 1790715076507675 us; error 46 us; skew 500 ppm
I20260929 20:51:16.508080 17506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:16 dist-test-slave-blv7 krb5kdc[17493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:16.511111 17506 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:51:16.512513 17506 webserver.cc:460] Webserver started at http://127.0.97.62:34571/ using document root <none> and password file <none>
I20260929 20:51:16.513186 17506 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:16.513298 17506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:16.513576 17506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:16.516795 17506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "2843096e182a452e8a6f7ae9e525292e"
format_stamp: "Formatted at 2026-09-29 20:51:16 on dist-test-slave-blv7"
server_key: "9bf7514dbd39d2353852801ebc328939"
server_key_iv: "cb6d694f258d5b877b7554f7f730f000"
server_key_version: "encryptionkey@0"
I20260929 20:51:16.517534 17506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "2843096e182a452e8a6f7ae9e525292e"
format_stamp: "Formatted at 2026-09-29 20:51:16 on dist-test-slave-blv7"
server_key: "9bf7514dbd39d2353852801ebc328939"
server_key_iv: "cb6d694f258d5b877b7554f7f730f000"
server_key_version: "encryptionkey@0"
I20260929 20:51:16.522928 17506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:51:16.526407 17521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:16.527884 17506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 20:51:16.528039 17506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "2843096e182a452e8a6f7ae9e525292e"
format_stamp: "Formatted at 2026-09-29 20:51:16 on dist-test-slave-blv7"
server_key: "9bf7514dbd39d2353852801ebc328939"
server_key_iv: "cb6d694f258d5b877b7554f7f730f000"
server_key_version: "encryptionkey@0"
I20260929 20:51:16.528215 17506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:16.541723 17506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:16.545722 17506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:16.545985 17506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:16.563362 17506 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46731
I20260929 20:51:16.563427 17572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46731 every 8 connection(s)
I20260929 20:51:16.564710 17506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:51:16.568320 17573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:16.572346 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 17506
I20260929 20:51:16.572595 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:51:16.573377 388 external_mini_cluster.cc:1428] Setting key b1dd7b679713f81f1278aa349618a313
I20260929 20:51:16.579448 17573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e: Bootstrap starting.
I20260929 20:51:16.582780 17573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:16.583993 17573 log.cc:824] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:16.587275 17573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e: No bootstrap required, opened a new log
Sep 29 20:51:16 dist-test-slave-blv7 krb5kdc[17493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:51:16.597189 17573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2843096e182a452e8a6f7ae9e525292e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46731 } }
I20260929 20:51:16.597674 17573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:16.597779 17573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2843096e182a452e8a6f7ae9e525292e, State: Initialized, Role: FOLLOWER
I20260929 20:51:16.598378 17576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:16.575449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.0.97.62:46731)
0929 20:51:16.576205 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:16.576217 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:16.576258 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:51:16.578235 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:16.578258 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:16.578645 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:16.578978 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:16.580666 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:16.582113 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:16.583949 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:16.584961 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:16.585164 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:16.589673 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:16.589684 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:16.589706 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:16.589768 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:16.593595 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:16.594334 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:16.594343 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:16.594420 (+ 77us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:16.594636 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:16.595078 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:16.595083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:16.595087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:16.595830 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:16.596130 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:16.596557 (+ 427us) server_negotiation.cc:299] Negotiation successful
0929 20:51:16.596991 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":332,"threads_started":1}
I20260929 20:51:16.598454 17573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2843096e182a452e8a6f7ae9e525292e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46731 } }
I20260929 20:51:16.598721 17573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:51:16.598870 17573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:51:16.599079 17573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:16.601688 17573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2843096e182a452e8a6f7ae9e525292e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46731 } }
I20260929 20:51:16.602131 17573 leader_election.cc:302] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2843096e182a452e8a6f7ae9e525292e; no voters:
I20260929 20:51:16.602784 17573 leader_election.cc:288] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:51:16.602905 17578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:16.603683 17578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e [term 1 LEADER]: Becoming Leader. State: Replica: 2843096e182a452e8a6f7ae9e525292e, State: Running, Role: LEADER
I20260929 20:51:16.604187 17578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2843096e182a452e8a6f7ae9e525292e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46731 } }
I20260929 20:51:16.604846 17573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:51:16.606952 17580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2843096e182a452e8a6f7ae9e525292e. Latest consensus state: current_term: 1 leader_uuid: "2843096e182a452e8a6f7ae9e525292e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2843096e182a452e8a6f7ae9e525292e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46731 } } }
I20260929 20:51:16.607343 17580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:16.608012 17579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2843096e182a452e8a6f7ae9e525292e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2843096e182a452e8a6f7ae9e525292e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46731 } } }
I20260929 20:51:16.608342 17579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:16.608927 17585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:51:16.613194 17585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:51:16.621363 17585 catalog_manager.cc:1347] Generated new cluster ID: d16b510b202c41d79cf426f909b3e2b2
I20260929 20:51:16.621483 17585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:51:16.648324 17585 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:51:16.650261 17585 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:51:16.661664 17585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e: Generated new TSK 0
I20260929 20:51:16.662488 17585 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:51:16.738189 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46731
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:17.061170 17599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:17.061441 17599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:17.061573 17599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:17.068585 17599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:17.068689 17599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:17.068830 17599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:51:17.078603 17599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:17.079883 17599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:17.081175 17599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:17.093431 17604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:17.093453 17607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:17.093511 17605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:17.094172 17599 server_base.cc:1034] running on GCE node
I20260929 20:51:17.094707 17599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:17.095621 17599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:17.096813 17599 hybrid_clock.cc:648] HybridClock initialized: now 1790715077096770 us; error 32 us; skew 500 ppm
I20260929 20:51:17.097188 17599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:17 dist-test-slave-blv7 krb5kdc[17493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:17.100191 17599 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:51:17.101490 17599 webserver.cc:460] Webserver started at http://127.0.97.1:39855/ using document root <none> and password file <none>
I20260929 20:51:17.102140 17599 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:17.102233 17599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:17.102466 17599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:17.105238 17599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "1e765389f20d48df8c13868dd55238e5"
format_stamp: "Formatted at 2026-09-29 20:51:17 on dist-test-slave-blv7"
server_key: "22832c885e573f78874a44656fc3b18a"
server_key_iv: "19cd301b0ec0ec2af3fa3c645ce5e42b"
server_key_version: "encryptionkey@0"
I20260929 20:51:17.106122 17599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1e765389f20d48df8c13868dd55238e5"
format_stamp: "Formatted at 2026-09-29 20:51:17 on dist-test-slave-blv7"
server_key: "22832c885e573f78874a44656fc3b18a"
server_key_iv: "19cd301b0ec0ec2af3fa3c645ce5e42b"
server_key_version: "encryptionkey@0"
I20260929 20:51:17.111188 17599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 20:51:17.114593 17614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:17.116006 17599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:51:17.116138 17599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "1e765389f20d48df8c13868dd55238e5"
format_stamp: "Formatted at 2026-09-29 20:51:17 on dist-test-slave-blv7"
server_key: "22832c885e573f78874a44656fc3b18a"
server_key_iv: "19cd301b0ec0ec2af3fa3c645ce5e42b"
server_key_version: "encryptionkey@0"
I20260929 20:51:17.116302 17599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:17.148138 17599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:17.152287 17599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:17.152555 17599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:17.153543 17599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:51:17.154909 17599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:51:17.154979 17599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:17.155094 17599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:51:17.155138 17599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:17.183712 17599 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45873
I20260929 20:51:17.183759 17726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45873 every 8 connection(s)
I20260929 20:51:17.184954 17599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:51:17.193630 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 17599
I20260929 20:51:17.193802 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:51:17.194413 388 external_mini_cluster.cc:1428] Setting key 08a906a2747d1552ad606e4f45e99ba0
Sep 29 20:51:17 dist-test-slave-blv7 krb5kdc[17493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:51:17.206209 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:17.187503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56987 (local address 127.0.97.62:46731)
0929 20:51:17.187921 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:17.187926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:17.188910 (+ 984us) server_negotiation.cc:407] Connection header received
0929 20:51:17.190699 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:17.190705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:17.190791 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:17.190901 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:17.192408 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:17.193136 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:17.194740 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:17.195302 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:17.195414 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:17.198664 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:17.198669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:17.198672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:17.198703 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:17.202309 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:17.203548 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:17.203554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:17.203558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:17.203742 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:17.204374 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:17.204379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:17.204382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:17.204706 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:17.204865 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:17.205613 (+ 748us) server_negotiation.cc:299] Negotiation successful
0929 20:51:17.205848 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":206,"threads_started":1}
I20260929 20:51:17.207234 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:17.188077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46731 (local address 127.0.97.1:56987)
0929 20:51:17.188752 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:17.188794 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:17.190431 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:17.191175 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:17.191184 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:17.191702 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:51:17.192243 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:17.192260 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:17.193290 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:17.193296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:17.194605 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:17.194617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:17.195533 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:17.195540 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:17.195764 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:17.196392 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:51:17.196421 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:51:17.198456 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:51:17.202523 (+ 4067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:51:17.202532 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:51:17.202573 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:51:17.203331 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:51:17.203917 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:51:17.203922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:51:17.203925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:51:17.204240 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:51:17.204839 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:51:17.204847 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:51:17.205228 (+ 381us) client_negotiation.cc:769] Sending connection context
0929 20:51:17.205513 (+ 285us) client_negotiation.cc:240] Negotiation successful
0929 20:51:17.206064 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":251,"threads_started":1}
I20260929 20:51:17.208762 17727 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46731
I20260929 20:51:17.209075 17727 heartbeater.cc:464] Registering TS with master...
I20260929 20:51:17.209870 17727 heartbeater.cc:511] Master 127.0.97.62:46731 requested a full tablet report, sending...
I20260929 20:51:17.212252 17538 ts_manager.cc:194] Registered new tserver with Master: 1e765389f20d48df8c13868dd55238e5 (127.0.97.1:45873)
I20260929 20:51:17.214170 17538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56987
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:51:17.247653 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46731
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:17.560039 17733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:17.560312 17733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:17.560406 17733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:17.567572 17733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:17.567664 17733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:17.567796 17733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:51:17.577186 17733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:17.578409 17733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:17.579692 17733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:17.595469 17741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:17.595470 17738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:17.595498 17739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:17.596222 17733 server_base.cc:1034] running on GCE node
I20260929 20:51:17.596835 17733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:17.597967 17733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:17.599172 17733 hybrid_clock.cc:648] HybridClock initialized: now 1790715077599115 us; error 45 us; skew 500 ppm
I20260929 20:51:17.599503 17733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:17 dist-test-slave-blv7 krb5kdc[17493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:17.602450 17733 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:51:17.603922 17733 webserver.cc:460] Webserver started at http://127.0.97.2:38501/ using document root <none> and password file <none>
I20260929 20:51:17.604568 17733 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:17.604663 17733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:17.604928 17733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:17.607890 17733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "05503942c17546dd9273f33be9c7d56f"
format_stamp: "Formatted at 2026-09-29 20:51:17 on dist-test-slave-blv7"
server_key: "cf9969938249fe857c186d3bcb158df0"
server_key_iv: "0eb9d9dfdb54236db127b436c26ec3e0"
server_key_version: "encryptionkey@0"
I20260929 20:51:17.608623 17733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "05503942c17546dd9273f33be9c7d56f"
format_stamp: "Formatted at 2026-09-29 20:51:17 on dist-test-slave-blv7"
server_key: "cf9969938249fe857c186d3bcb158df0"
server_key_iv: "0eb9d9dfdb54236db127b436c26ec3e0"
server_key_version: "encryptionkey@0"
I20260929 20:51:17.613507 17733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:51:17.617242 17748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:17.618642 17733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:51:17.618784 17733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "05503942c17546dd9273f33be9c7d56f"
format_stamp: "Formatted at 2026-09-29 20:51:17 on dist-test-slave-blv7"
server_key: "cf9969938249fe857c186d3bcb158df0"
server_key_iv: "0eb9d9dfdb54236db127b436c26ec3e0"
server_key_version: "encryptionkey@0"
I20260929 20:51:17.618965 17733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:17.642406 17733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:17.646569 17733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:17.646822 17733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:17.647886 17733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:51:17.649269 17733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:51:17.649341 17733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:17.649443 17733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:51:17.649518 17733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:17.679553 17733 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33619
I20260929 20:51:17.679623 17860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33619 every 8 connection(s)
I20260929 20:51:17.680788 17733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:51:17.689428 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 17733
I20260929 20:51:17.689626 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:51:17.690158 388 external_mini_cluster.cc:1428] Setting key e5b343b9a863d4af56324711e13fa7da
Sep 29 20:51:17 dist-test-slave-blv7 krb5kdc[17493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:51:17.701884 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:17.683425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45729 (local address 127.0.97.62:46731)
0929 20:51:17.683634 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:17.683641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:17.684774 (+ 1133us) server_negotiation.cc:407] Connection header received
0929 20:51:17.686523 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:17.686528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:17.686599 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:17.686691 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:17.688381 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:17.689158 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:17.691336 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:17.691839 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:17.691953 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:17.695653 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:17.695659 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:17.695664 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:17.695703 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:17.698670 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:17.699449 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:17.699454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:17.699458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:17.699600 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:17.700083 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:17.700087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:17.700090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:17.700397 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:17.700510 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:17.701445 (+ 935us) server_negotiation.cc:299] Negotiation successful
0929 20:51:17.701632 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:51:17.703155 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:17.683909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46731 (local address 127.0.97.2:45729)
0929 20:51:17.684617 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:17.684658 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:17.686259 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:17.686883 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:17.686893 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:17.687592 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:51:17.688155 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:17.688174 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:17.689354 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:17.689360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:17.691199 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:17.691212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:17.692074 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:17.692082 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:17.692369 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:17.693105 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:51:17.693129 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:51:17.695425 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:51:17.698817 (+ 3392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:51:17.698825 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:51:17.698852 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:51:17.699332 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:51:17.699708 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:51:17.699713 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:51:17.699717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:51:17.699967 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:51:17.700520 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:51:17.700528 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:51:17.700967 (+ 439us) client_negotiation.cc:769] Sending connection context
0929 20:51:17.701312 (+ 345us) client_negotiation.cc:240] Negotiation successful
0929 20:51:17.701816 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":256,"threads_started":1}
I20260929 20:51:17.704617 17861 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46731
I20260929 20:51:17.704916 17861 heartbeater.cc:464] Registering TS with master...
I20260929 20:51:17.705683 17861 heartbeater.cc:511] Master 127.0.97.62:46731 requested a full tablet report, sending...
I20260929 20:51:17.707245 17538 ts_manager.cc:194] Registered new tserver with Master: 05503942c17546dd9273f33be9c7d56f (127.0.97.2:33619)
I20260929 20:51:17.708123 17538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45729
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:51:17.742812 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46731
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:18.061309 17866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:18.061619 17866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:18.061744 17866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:18.068863 17866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:18.068976 17866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:18.069128 17866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:51:18.078720 17866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:18.079990 17866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:18.081306 17866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:18.093540 17874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:18.093582 17872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:18.093719 17871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:18.094205 17866 server_base.cc:1034] running on GCE node
I20260929 20:51:18.094779 17866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:18.095778 17866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:18.096990 17866 hybrid_clock.cc:648] HybridClock initialized: now 1790715078096942 us; error 39 us; skew 500 ppm
I20260929 20:51:18.097285 17866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:18 dist-test-slave-blv7 krb5kdc[17493](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:18.100345 17866 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:51:18.101818 17866 webserver.cc:460] Webserver started at http://127.0.97.3:44435/ using document root <none> and password file <none>
I20260929 20:51:18.102474 17866 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:18.102586 17866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:18.102866 17866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:18.105772 17866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "333bef8dc6764b41b28291c0387c4f81"
format_stamp: "Formatted at 2026-09-29 20:51:18 on dist-test-slave-blv7"
server_key: "be5d1d57217c225525c8080e0c1c6e3c"
server_key_iv: "87060e0234696252161fffc88cb11458"
server_key_version: "encryptionkey@0"
I20260929 20:51:18.106555 17866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "333bef8dc6764b41b28291c0387c4f81"
format_stamp: "Formatted at 2026-09-29 20:51:18 on dist-test-slave-blv7"
server_key: "be5d1d57217c225525c8080e0c1c6e3c"
server_key_iv: "87060e0234696252161fffc88cb11458"
server_key_version: "encryptionkey@0"
I20260929 20:51:18.111455 17866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 20:51:18.114732 17881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:18.116014 17866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:51:18.116163 17866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "333bef8dc6764b41b28291c0387c4f81"
format_stamp: "Formatted at 2026-09-29 20:51:18 on dist-test-slave-blv7"
server_key: "be5d1d57217c225525c8080e0c1c6e3c"
server_key_iv: "87060e0234696252161fffc88cb11458"
server_key_version: "encryptionkey@0"
I20260929 20:51:18.116346 17866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:18.142786 17866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:18.146903 17866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:18.147219 17866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:18.148197 17866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:51:18.149540 17866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:51:18.149616 17866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:18.149724 17866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:51:18.149775 17866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:18.180280 17866 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37999
I20260929 20:51:18.180344 17993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37999 every 8 connection(s)
I20260929 20:51:18.181509 17866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:51:18.185442 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 17866
I20260929 20:51:18.185631 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:51:18.186170 388 external_mini_cluster.cc:1428] Setting key 9477377d0b56087f0fe2222426364416
Sep 29 20:51:18 dist-test-slave-blv7 krb5kdc[17493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:51:18.201496 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:18.184035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53797 (local address 127.0.97.62:46731)
0929 20:51:18.184195 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:18.184199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:18.185718 (+ 1519us) server_negotiation.cc:407] Connection header received
0929 20:51:18.188143 (+ 2425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:18.188149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:18.188233 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:18.188318 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:18.189988 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.190716 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.192282 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.192659 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.192764 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:18.195941 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:18.195946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:18.195950 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:18.195981 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:18.198436 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:18.199118 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:18.199122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:18.199124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:18.199250 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:18.199696 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:18.199699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:18.199701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:18.199962 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:18.200096 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:18.200935 (+ 839us) server_negotiation.cc:299] Negotiation successful
0929 20:51:18.201155 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:51:18.202374 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:18.184678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46731 (local address 127.0.97.3:53797)
0929 20:51:18.185538 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:18.185586 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:18.187866 (+ 2280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:18.188580 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:18.188591 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:18.189293 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:51:18.189826 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:18.189843 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.190864 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.190870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:18.192128 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:18.192144 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.192870 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.192877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:18.193093 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:18.193693 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:51:18.193714 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:51:18.195732 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:51:18.198567 (+ 2835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:51:18.198576 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:51:18.198590 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:51:18.198974 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:51:18.199408 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:51:18.199411 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:51:18.199413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:51:18.199581 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:51:18.200084 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:51:18.200092 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:51:18.200472 (+ 380us) client_negotiation.cc:769] Sending connection context
0929 20:51:18.200759 (+ 287us) client_negotiation.cc:240] Negotiation successful
0929 20:51:18.201250 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":328,"threads_started":1}
I20260929 20:51:18.203699 17994 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46731
I20260929 20:51:18.203976 17994 heartbeater.cc:464] Registering TS with master...
I20260929 20:51:18.204689 17994 heartbeater.cc:511] Master 127.0.97.62:46731 requested a full tablet report, sending...
I20260929 20:51:18.206037 17538 ts_manager.cc:194] Registered new tserver with Master: 333bef8dc6764b41b28291c0387c4f81 (127.0.97.3:37999)
I20260929 20:51:18.206862 17538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53797
I20260929 20:51:18.214098 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:51:18.217243 17727 heartbeater.cc:503] Master 127.0.97.62:46731 was elected leader, sending a full tablet report...
I20260929 20:51:18.231109 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:18.218616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36948 (local address 127.0.97.62:46731)
0929 20:51:18.218791 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:18.218796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:18.219274 (+ 478us) server_negotiation.cc:407] Connection header received
0929 20:51:18.219415 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:18.219419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:18.219476 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:18.219552 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:18.221085 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.221818 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.223120 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.223609 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.223704 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:18.225954 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:18.225959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:18.225962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:18.225992 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:18.228566 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:18.229222 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:18.229226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:18.229228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:18.229389 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:18.229702 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:18.229705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:18.229706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:18.229992 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:18.230178 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:18.230560 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 20:51:18.230758 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:51:18.235262 17538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:51:18.241966 17538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:51:18.271934 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:18.253826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37999 (local address 127.0.0.1:33698)
0929 20:51:18.254517 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:18.254569 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:18.254736 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:18.264460 (+ 9724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:18.264468 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:18.264523 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:18.264932 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:18.264944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.266997 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.267002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:18.269427 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:18.269439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.270851 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.270857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:18.271123 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:18.271187 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 20:51:18.271295 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 20:51:18.271472 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":115,"spinlock_wait_cycles":2304,"thread_start_us":254,"threads_started":1}
I20260929 20:51:18.273062 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:18.254940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33619 (local address 127.0.0.1:45390)
0929 20:51:18.264374 (+ 9434us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:18.264392 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:18.264544 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:18.266302 (+ 1758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:18.266306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:18.266332 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:18.266654 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:18.266665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.268347 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.268352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:18.270494 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:18.270507 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.272268 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.272275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:18.272445 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:18.272480 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 20:51:18.272551 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 20:51:18.272677 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9282,"mutex_wait_us":53,"spinlock_wait_cycles":28288,"thread_start_us":9318,"threads_started":1}
I20260929 20:51:18.273284 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:18.254193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33698 (local address 127.0.97.3:37999)
0929 20:51:18.263733 (+ 9540us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:18.263742 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:18.263784 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:51:18.263900 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:18.263908 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:18.264196 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:18.265129 (+ 933us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:18.265433 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.266863 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.269583 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.270636 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.272485 (+ 1849us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:18.272718 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:18.272788 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 20:51:18.272961 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9362,"thread_start_us":9209,"threads_started":1}
I20260929 20:51:18.273958 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:18.254309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45873 (local address 127.0.0.1:55128)
0929 20:51:18.264007 (+ 9698us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:18.264025 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:18.264159 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:18.265713 (+ 1554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:18.265718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:18.265742 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:18.266070 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:18.266079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.267920 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.267925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:18.270117 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:18.270132 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.273195 (+ 3063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.273201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:18.273365 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:18.273401 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 20:51:18.273487 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 20:51:18.273608 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9578,"mutex_wait_us":1,"thread_start_us":9449,"threads_started":1}
I20260929 20:51:18.274225 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:18.255227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45390 (local address 127.0.97.2:33619)
0929 20:51:18.265720 (+ 10493us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:18.265729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:18.265754 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:51:18.265855 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:18.265862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:18.266076 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:18.266290 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:18.266847 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.268198 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.270665 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.272040 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.273470 (+ 1430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:18.273668 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:18.273735 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 20:51:18.273913 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10339,"thread_start_us":10219,"threads_started":1}
I20260929 20:51:18.275800 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:18.254963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55128 (local address 127.0.97.1:45873)
0929 20:51:18.265098 (+ 10135us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:18.265107 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:18.265139 (+ 32us) server_negotiation.cc:407] Connection header received
0929 20:51:18.265239 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:18.265247 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:18.265529 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:18.265714 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:18.266221 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.267759 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.270319 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.272761 (+ 2442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.272874 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:18.273075 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:18.274144 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0929 20:51:18.274301 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9969,"thread_start_us":9817,"threads_started":1}
I20260929 20:51:18.277633 17929 tablet_service.cc:1467] Processing CreateTablet for tablet 558c63883a9f446a9255f3ef8bf6386e (DEFAULT_TABLE table=test-table [id=73c16f4be2b74f6fba2a195a1459313c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:51:18.279160 17929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558c63883a9f446a9255f3ef8bf6386e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:18.279642 17662 tablet_service.cc:1467] Processing CreateTablet for tablet 558c63883a9f446a9255f3ef8bf6386e (DEFAULT_TABLE table=test-table [id=73c16f4be2b74f6fba2a195a1459313c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:51:18.280939 17662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558c63883a9f446a9255f3ef8bf6386e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:18.280452 17796 tablet_service.cc:1467] Processing CreateTablet for tablet 558c63883a9f446a9255f3ef8bf6386e (DEFAULT_TABLE table=test-table [id=73c16f4be2b74f6fba2a195a1459313c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:51:18.281742 17796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558c63883a9f446a9255f3ef8bf6386e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:18.292439 18012 tablet_bootstrap.cc:492] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5: Bootstrap starting.
I20260929 20:51:18.293743 18013 tablet_bootstrap.cc:492] T 558c63883a9f446a9255f3ef8bf6386e P 333bef8dc6764b41b28291c0387c4f81: Bootstrap starting.
I20260929 20:51:18.296008 18014 tablet_bootstrap.cc:492] T 558c63883a9f446a9255f3ef8bf6386e P 05503942c17546dd9273f33be9c7d56f: Bootstrap starting.
I20260929 20:51:18.297212 18012 tablet_bootstrap.cc:654] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:18.297374 18013 tablet_bootstrap.cc:654] T 558c63883a9f446a9255f3ef8bf6386e P 333bef8dc6764b41b28291c0387c4f81: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:18.298496 18013 log.cc:824] T 558c63883a9f446a9255f3ef8bf6386e P 333bef8dc6764b41b28291c0387c4f81: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:18.298497 18012 log.cc:824] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:18.298635 18014 tablet_bootstrap.cc:654] T 558c63883a9f446a9255f3ef8bf6386e P 05503942c17546dd9273f33be9c7d56f: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:18.299758 18014 log.cc:824] T 558c63883a9f446a9255f3ef8bf6386e P 05503942c17546dd9273f33be9c7d56f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:18.301568 18013 tablet_bootstrap.cc:492] T 558c63883a9f446a9255f3ef8bf6386e P 333bef8dc6764b41b28291c0387c4f81: No bootstrap required, opened a new log
I20260929 20:51:18.301584 18012 tablet_bootstrap.cc:492] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5: No bootstrap required, opened a new log
I20260929 20:51:18.301810 18012 ts_tablet_manager.cc:1397] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260929 20:51:18.301836 18013 ts_tablet_manager.cc:1397] T 558c63883a9f446a9255f3ef8bf6386e P 333bef8dc6764b41b28291c0387c4f81: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 20:51:18.302889 18014 tablet_bootstrap.cc:492] T 558c63883a9f446a9255f3ef8bf6386e P 05503942c17546dd9273f33be9c7d56f: No bootstrap required, opened a new log
I20260929 20:51:18.303427 18014 ts_tablet_manager.cc:1397] T 558c63883a9f446a9255f3ef8bf6386e P 05503942c17546dd9273f33be9c7d56f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260929 20:51:18.309274 18012 raft_consensus.cc:348] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333bef8dc6764b41b28291c0387c4f81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "05503942c17546dd9273f33be9c7d56f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } } peers { permanent_uuid: "1e765389f20d48df8c13868dd55238e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45873 } }
I20260929 20:51:18.309787 18012 raft_consensus.cc:374] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:18.309947 18012 raft_consensus.cc:729] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e765389f20d48df8c13868dd55238e5, State: Initialized, Role: FOLLOWER
I20260929 20:51:18.310551 18012 consensus_queue.cc:260] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333bef8dc6764b41b28291c0387c4f81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "05503942c17546dd9273f33be9c7d56f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } } peers { permanent_uuid: "1e765389f20d48df8c13868dd55238e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45873 } }
I20260929 20:51:18.312112 18013 raft_consensus.cc:348] T 558c63883a9f446a9255f3ef8bf6386e P 333bef8dc6764b41b28291c0387c4f81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333bef8dc6764b41b28291c0387c4f81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "05503942c17546dd9273f33be9c7d56f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } } peers { permanent_uuid: "1e765389f20d48df8c13868dd55238e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45873 } }
I20260929 20:51:18.312528 18013 raft_consensus.cc:374] T 558c63883a9f446a9255f3ef8bf6386e P 333bef8dc6764b41b28291c0387c4f81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:18.312623 18013 raft_consensus.cc:729] T 558c63883a9f446a9255f3ef8bf6386e P 333bef8dc6764b41b28291c0387c4f81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333bef8dc6764b41b28291c0387c4f81, State: Initialized, Role: FOLLOWER
I20260929 20:51:18.313200 18013 consensus_queue.cc:260] T 558c63883a9f446a9255f3ef8bf6386e P 333bef8dc6764b41b28291c0387c4f81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333bef8dc6764b41b28291c0387c4f81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "05503942c17546dd9273f33be9c7d56f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } } peers { permanent_uuid: "1e765389f20d48df8c13868dd55238e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45873 } }
I20260929 20:51:18.313639 18012 ts_tablet_manager.cc:1428] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260929 20:51:18.314604 17994 heartbeater.cc:503] Master 127.0.97.62:46731 was elected leader, sending a full tablet report...
I20260929 20:51:18.315294 18013 ts_tablet_manager.cc:1428] T 558c63883a9f446a9255f3ef8bf6386e P 333bef8dc6764b41b28291c0387c4f81: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20260929 20:51:18.315158 18014 raft_consensus.cc:348] T 558c63883a9f446a9255f3ef8bf6386e P 05503942c17546dd9273f33be9c7d56f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333bef8dc6764b41b28291c0387c4f81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "05503942c17546dd9273f33be9c7d56f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } } peers { permanent_uuid: "1e765389f20d48df8c13868dd55238e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45873 } }
I20260929 20:51:18.315693 18014 raft_consensus.cc:374] T 558c63883a9f446a9255f3ef8bf6386e P 05503942c17546dd9273f33be9c7d56f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:18.315804 18014 raft_consensus.cc:729] T 558c63883a9f446a9255f3ef8bf6386e P 05503942c17546dd9273f33be9c7d56f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05503942c17546dd9273f33be9c7d56f, State: Initialized, Role: FOLLOWER
I20260929 20:51:18.316555 18014 consensus_queue.cc:260] T 558c63883a9f446a9255f3ef8bf6386e P 05503942c17546dd9273f33be9c7d56f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333bef8dc6764b41b28291c0387c4f81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "05503942c17546dd9273f33be9c7d56f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } } peers { permanent_uuid: "1e765389f20d48df8c13868dd55238e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45873 } }
I20260929 20:51:18.317729 17861 heartbeater.cc:503] Master 127.0.97.62:46731 was elected leader, sending a full tablet report...
I20260929 20:51:18.318114 18014 ts_tablet_manager.cc:1428] T 558c63883a9f446a9255f3ef8bf6386e P 05503942c17546dd9273f33be9c7d56f: Time spent starting tablet: real 0.015s user 0.006s sys 0.006s
W20260929 20:51:18.432688 17862 tablet.cc:2369] T 558c63883a9f446a9255f3ef8bf6386e P 05503942c17546dd9273f33be9c7d56f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:51:18.433099 17995 tablet.cc:2369] T 558c63883a9f446a9255f3ef8bf6386e P 333bef8dc6764b41b28291c0387c4f81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:51:18.433524 18018 raft_consensus.cc:482] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:51:18.433826 18018 raft_consensus.cc:504] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333bef8dc6764b41b28291c0387c4f81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "05503942c17546dd9273f33be9c7d56f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } } peers { permanent_uuid: "1e765389f20d48df8c13868dd55238e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45873 } }
I20260929 20:51:18.435523 18018 leader_election.cc:288] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 333bef8dc6764b41b28291c0387c4f81 (127.0.97.3:37999), 05503942c17546dd9273f33be9c7d56f (127.0.97.2:33619)
W20260929 20:51:18.437299 17728 tablet.cc:2369] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:51:18.441944 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:18.435875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37999 (local address 127.0.97.1:43633)
0929 20:51:18.436491 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:18.436506 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:18.436660 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:18.437114 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:18.437118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:18.437152 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:18.437537 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:18.437550 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.438561 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.438565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:18.440042 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:18.440054 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.441175 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.441181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:18.441340 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:18.441367 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 20:51:18.441456 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 20:51:18.441583 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":331,"threads_started":1}
I20260929 20:51:18.442358 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:18.435846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43633 (local address 127.0.97.3:37999)
0929 20:51:18.436059 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:18.436065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:18.436621 (+ 556us) server_negotiation.cc:407] Connection header received
0929 20:51:18.436816 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:18.436823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:18.436909 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:18.437001 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:18.437679 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.438425 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.440198 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.440976 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.441118 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:18.441220 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:18.441521 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 20:51:18.441770 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:51:18.442920 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:18.436815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54207 (local address 127.0.97.2:33619)
0929 20:51:18.437037 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:18.437043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:18.437384 (+ 341us) server_negotiation.cc:407] Connection header received
0929 20:51:18.437564 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:18.437569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:18.437644 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:18.437703 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:18.438395 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.439108 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.440751 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.441797 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.441930 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:18.442023 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:18.442542 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 20:51:18.442679 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:51:18.443636 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:18.436593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33619 (local address 127.0.97.1:54207)
0929 20:51:18.437223 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:18.437242 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:18.437375 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:18.437905 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:18.437909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:18.437933 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:18.438239 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:18.438249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.439238 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.439242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:18.440602 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:18.440613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.442177 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.442184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:18.442362 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:18.442396 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:51:18.443171 (+ 775us) client_negotiation.cc:240] Negotiation successful
0929 20:51:18.443305 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":303,"threads_started":1}
I20260929 20:51:18.444669 17949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558c63883a9f446a9255f3ef8bf6386e" candidate_uuid: "1e765389f20d48df8c13868dd55238e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333bef8dc6764b41b28291c0387c4f81" is_pre_election: true
I20260929 20:51:18.444679 17816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558c63883a9f446a9255f3ef8bf6386e" candidate_uuid: "1e765389f20d48df8c13868dd55238e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05503942c17546dd9273f33be9c7d56f" is_pre_election: true
I20260929 20:51:18.445212 17949 raft_consensus.cc:2454] T 558c63883a9f446a9255f3ef8bf6386e P 333bef8dc6764b41b28291c0387c4f81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e765389f20d48df8c13868dd55238e5 in term 0.
I20260929 20:51:18.445211 17816 raft_consensus.cc:2454] T 558c63883a9f446a9255f3ef8bf6386e P 05503942c17546dd9273f33be9c7d56f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e765389f20d48df8c13868dd55238e5 in term 0.
I20260929 20:51:18.445958 17615 leader_election.cc:302] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e765389f20d48df8c13868dd55238e5, 333bef8dc6764b41b28291c0387c4f81; no voters:
I20260929 20:51:18.446414 18018 raft_consensus.cc:2781] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:51:18.446578 18018 raft_consensus.cc:482] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:51:18.446683 18018 raft_consensus.cc:3037] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:18.449317 18018 raft_consensus.cc:504] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333bef8dc6764b41b28291c0387c4f81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "05503942c17546dd9273f33be9c7d56f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } } peers { permanent_uuid: "1e765389f20d48df8c13868dd55238e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45873 } }
I20260929 20:51:18.449903 18018 leader_election.cc:288] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5 [CANDIDATE]: Term 1 election: Requested vote from peers 333bef8dc6764b41b28291c0387c4f81 (127.0.97.3:37999), 05503942c17546dd9273f33be9c7d56f (127.0.97.2:33619)
I20260929 20:51:18.450461 17949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558c63883a9f446a9255f3ef8bf6386e" candidate_uuid: "1e765389f20d48df8c13868dd55238e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333bef8dc6764b41b28291c0387c4f81"
I20260929 20:51:18.450527 17816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558c63883a9f446a9255f3ef8bf6386e" candidate_uuid: "1e765389f20d48df8c13868dd55238e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05503942c17546dd9273f33be9c7d56f"
I20260929 20:51:18.450652 17949 raft_consensus.cc:3037] T 558c63883a9f446a9255f3ef8bf6386e P 333bef8dc6764b41b28291c0387c4f81 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:18.450729 17816 raft_consensus.cc:3037] T 558c63883a9f446a9255f3ef8bf6386e P 05503942c17546dd9273f33be9c7d56f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:18.452895 17816 raft_consensus.cc:2454] T 558c63883a9f446a9255f3ef8bf6386e P 05503942c17546dd9273f33be9c7d56f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e765389f20d48df8c13868dd55238e5 in term 1.
I20260929 20:51:18.452920 17949 raft_consensus.cc:2454] T 558c63883a9f446a9255f3ef8bf6386e P 333bef8dc6764b41b28291c0387c4f81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e765389f20d48df8c13868dd55238e5 in term 1.
I20260929 20:51:18.453275 17615 leader_election.cc:302] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05503942c17546dd9273f33be9c7d56f, 1e765389f20d48df8c13868dd55238e5; no voters:
I20260929 20:51:18.453537 18018 raft_consensus.cc:2781] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:18.453963 18018 raft_consensus.cc:686] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5 [term 1 LEADER]: Becoming Leader. State: Replica: 1e765389f20d48df8c13868dd55238e5, State: Running, Role: LEADER
I20260929 20:51:18.454725 18018 consensus_queue.cc:237] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333bef8dc6764b41b28291c0387c4f81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } } peers { permanent_uuid: "05503942c17546dd9273f33be9c7d56f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } } peers { permanent_uuid: "1e765389f20d48df8c13868dd55238e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45873 } }
I20260929 20:51:18.459546 17538 catalog_manager.cc:5495] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e765389f20d48df8c13868dd55238e5 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "1e765389f20d48df8c13868dd55238e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333bef8dc6764b41b28291c0387c4f81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05503942c17546dd9273f33be9c7d56f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e765389f20d48df8c13868dd55238e5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45873 } health_report { overall_health: HEALTHY } } }
I20260929 20:51:18.513549 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:18.508277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55144 (local address 127.0.97.1:45873)
0929 20:51:18.508457 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:18.508462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:18.508478 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:51:18.508567 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:18.508570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:18.508643 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:18.508764 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:51:18.509106 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.510068 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.511576 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.511966 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.512060 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:18.512563 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:51:18.513072 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:51:18.513178 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:18.513284 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 20:51:18.513347 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:51:18.522212 17949 raft_consensus.cc:1261] T 558c63883a9f446a9255f3ef8bf6386e P 333bef8dc6764b41b28291c0387c4f81 [term 1 FOLLOWER]: Refusing update from remote peer 1e765389f20d48df8c13868dd55238e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:51:18.522212 17816 raft_consensus.cc:1261] T 558c63883a9f446a9255f3ef8bf6386e P 05503942c17546dd9273f33be9c7d56f [term 1 FOLLOWER]: Refusing update from remote peer 1e765389f20d48df8c13868dd55238e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:51:18.523273 18023 consensus_queue.cc:1035] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "333bef8dc6764b41b28291c0387c4f81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:51:18.523670 18018 consensus_queue.cc:1035] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05503942c17546dd9273f33be9c7d56f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:51:18.549232 17537 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:51:18.549494 17537 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:51:18.553006 17537 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2843096e182a452e8a6f7ae9e525292e: Sending DeleteTablet for 3 replicas of tablet 558c63883a9f446a9255f3ef8bf6386e
I20260929 20:51:18.554244 17662 tablet_service.cc:1514] Processing DeleteTablet for tablet 558c63883a9f446a9255f3ef8bf6386e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:51:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55128
I20260929 20:51:18.554273 17929 tablet_service.cc:1514] Processing DeleteTablet for tablet 558c63883a9f446a9255f3ef8bf6386e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:51:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33698
I20260929 20:51:18.554576 17796 tablet_service.cc:1514] Processing DeleteTablet for tablet 558c63883a9f446a9255f3ef8bf6386e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:51:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45390
I20260929 20:51:18.555354 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 17599
I20260929 20:51:18.573725 18038 tablet_replica.cc:330] T 558c63883a9f446a9255f3ef8bf6386e P 333bef8dc6764b41b28291c0387c4f81: stopping tablet replica
I20260929 20:51:18.574172 18038 raft_consensus.cc:2229] T 558c63883a9f446a9255f3ef8bf6386e P 333bef8dc6764b41b28291c0387c4f81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:51:18.574434 18037 tablet_replica.cc:330] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5: stopping tablet replica
I20260929 20:51:18.574704 18038 raft_consensus.cc:2258] T 558c63883a9f446a9255f3ef8bf6386e P 333bef8dc6764b41b28291c0387c4f81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:51:18.574859 18037 raft_consensus.cc:2229] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:51:18.575516 18037 raft_consensus.cc:2258] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:51:18.577746 18037 ts_tablet_manager.cc:1905] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 20:51:18 dist-test-slave-blv7 krb5kdc[17493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:51:18.580327 18038 ts_tablet_manager.cc:1905] T 558c63883a9f446a9255f3ef8bf6386e P 333bef8dc6764b41b28291c0387c4f81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:51:18.583488 18039 tablet_replica.cc:330] T 558c63883a9f446a9255f3ef8bf6386e P 05503942c17546dd9273f33be9c7d56f: stopping tablet replica
I20260929 20:51:18.583890 18039 raft_consensus.cc:2229] T 558c63883a9f446a9255f3ef8bf6386e P 05503942c17546dd9273f33be9c7d56f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:51:18.584352 18039 raft_consensus.cc:2258] T 558c63883a9f446a9255f3ef8bf6386e P 05503942c17546dd9273f33be9c7d56f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:51:18.584980 18037 ts_tablet_manager.cc:1918] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:51:18.585145 18037 log.cc:1196] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/558c63883a9f446a9255f3ef8bf6386e
I20260929 20:51:18.585669 18037 ts_tablet_manager.cc:1939] T 558c63883a9f446a9255f3ef8bf6386e P 1e765389f20d48df8c13868dd55238e5: Deleting consensus metadata
I20260929 20:51:18.587975 17524 catalog_manager.cc:4841] TS 1e765389f20d48df8c13868dd55238e5 (127.0.97.1:45873): tablet 558c63883a9f446a9255f3ef8bf6386e (table test-table [id=73c16f4be2b74f6fba2a195a1459313c]) successfully deleted
I20260929 20:51:18.589584 18039 ts_tablet_manager.cc:1905] T 558c63883a9f446a9255f3ef8bf6386e P 05503942c17546dd9273f33be9c7d56f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:51:18.592131 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:18.557833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55146 (local address 127.0.97.1:45873)
0929 20:51:18.558256 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:18.558263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:18.566479 (+ 8216us) server_negotiation.cc:407] Connection header received
0929 20:51:18.566872 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:18.566879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:18.566960 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:18.567055 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:18.568721 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.569890 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.571772 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.572454 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.572546 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:18.584777 (+ 12231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:18.584792 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:18.584819 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:18.584875 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:18.588691 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:18.589559 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:18.589568 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:18.589572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:18.589752 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:18.590226 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:18.590232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:18.590235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:18.590642 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:18.590926 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:18.591516 (+ 590us) server_negotiation.cc:299] Negotiation successful
0929 20:51:18.591750 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297}
I20260929 20:51:18.592463 18038 ts_tablet_manager.cc:1918] T 558c63883a9f446a9255f3ef8bf6386e P 333bef8dc6764b41b28291c0387c4f81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:51:18.592618 18038 log.cc:1196] T 558c63883a9f446a9255f3ef8bf6386e P 333bef8dc6764b41b28291c0387c4f81: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/558c63883a9f446a9255f3ef8bf6386e
I20260929 20:51:18.593107 18038 ts_tablet_manager.cc:1939] T 558c63883a9f446a9255f3ef8bf6386e P 333bef8dc6764b41b28291c0387c4f81: Deleting consensus metadata
I20260929 20:51:18.593570 17722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:18.594985 17522 catalog_manager.cc:4841] TS 333bef8dc6764b41b28291c0387c4f81 (127.0.97.3:37999): tablet 558c63883a9f446a9255f3ef8bf6386e (table test-table [id=73c16f4be2b74f6fba2a195a1459313c]) successfully deleted
I20260929 20:51:18.597034 18039 ts_tablet_manager.cc:1918] T 558c63883a9f446a9255f3ef8bf6386e P 05503942c17546dd9273f33be9c7d56f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:51:18.597188 18039 log.cc:1196] T 558c63883a9f446a9255f3ef8bf6386e P 05503942c17546dd9273f33be9c7d56f: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/558c63883a9f446a9255f3ef8bf6386e
I20260929 20:51:18.597687 18039 ts_tablet_manager.cc:1939] T 558c63883a9f446a9255f3ef8bf6386e P 05503942c17546dd9273f33be9c7d56f: Deleting consensus metadata
I20260929 20:51:18.599146 17522 catalog_manager.cc:4841] TS 05503942c17546dd9273f33be9c7d56f (127.0.97.2:33619): tablet 558c63883a9f446a9255f3ef8bf6386e (table test-table [id=73c16f4be2b74f6fba2a195a1459313c]) successfully deleted
I20260929 20:51:18.695245 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 17599
I20260929 20:51:18.710253 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 17733
Sep 29 20:51:18 dist-test-slave-blv7 krb5kdc[17493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:51:18.724924 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:18.711008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45402 (local address 127.0.97.2:33619)
0929 20:51:18.711198 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:18.711202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:18.711220 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:51:18.711383 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:18.711387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:18.711494 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:18.711575 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:18.712797 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.713523 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.714755 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.715194 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.715284 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:18.719248 (+ 3964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:18.719261 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:18.719275 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:18.719310 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:18.722084 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:18.722848 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:18.722854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:18.722856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:18.722972 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:18.723372 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:18.723375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:18.723377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:18.723789 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:18.724059 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:18.724385 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 20:51:18.724597 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:51:18.725553 17856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:18.812952 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 17733
I20260929 20:51:18.826761 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 17866
Sep 29 20:51:18 dist-test-slave-blv7 krb5kdc[17493](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:51:18.841631 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:18.827503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.0.97.3:37999)
0929 20:51:18.827687 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:18.827691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:18.827709 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:51:18.827801 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:18.827820 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:18.827885 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:18.827964 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:18.829211 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.829938 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.831439 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:18.831871 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:18.831962 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:18.836380 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:18.836395 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:18.836412 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:18.836452 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:18.839000 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:18.839768 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:18.839775 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:18.839777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:18.839939 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:18.840354 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:18.840358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:18.840360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:18.840685 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:18.840927 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:18.841194 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 20:51:18.841357 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:51:18.842288 17989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:18.928265 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 17866
I20260929 20:51:18.941613 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 17506
I20260929 20:51:18.942746 17568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:19.049140 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 17506
2026-09-29T20:51:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4131 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T20:51:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:51:19Z Disabled control of system clock
I20260929 20:51:19.115634 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36067
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:51:19.430212 18048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:19.430495 18048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:19.430599 18048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:19.437793 18048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:51:19.437891 18048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:19.437947 18048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:19.438025 18048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:51:19.438089 18048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:51:19.447393 18048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36067
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:19.448658 18048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:19.450398 18048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:19.461869 18053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:19.461913 18054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:19.461937 18056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:19.462579 18048 server_base.cc:1034] running on GCE node
I20260929 20:51:19.463418 18048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:19.464825 18048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:19.466042 18048 hybrid_clock.cc:648] HybridClock initialized: now 1790715079465982 us; error 50 us; skew 500 ppm
I20260929 20:51:19.466392 18048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:51:19.474852 18048 webserver.cc:460] Webserver started at http://127.0.97.62:41419/ using document root <none> and password file <none>
I20260929 20:51:19.475570 18048 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:19.475672 18048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:19.475972 18048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:19.478792 18048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "c11d8edc0155470f833daf75986202e0"
format_stamp: "Formatted at 2026-09-29 20:51:19 on dist-test-slave-blv7"
server_key: "a41dc1fb5096fcfeeab969da762ceb83"
server_key_iv: "6e5283ada65f088f8eedd5e3e3270961"
server_key_version: "encryptionkey@0"
I20260929 20:51:19.479598 18048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "c11d8edc0155470f833daf75986202e0"
format_stamp: "Formatted at 2026-09-29 20:51:19 on dist-test-slave-blv7"
server_key: "a41dc1fb5096fcfeeab969da762ceb83"
server_key_iv: "6e5283ada65f088f8eedd5e3e3270961"
server_key_version: "encryptionkey@0"
I20260929 20:51:19.484598 18048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:51:19.487954 18062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:19.489246 18048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:51:19.489408 18048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "c11d8edc0155470f833daf75986202e0"
format_stamp: "Formatted at 2026-09-29 20:51:19 on dist-test-slave-blv7"
server_key: "a41dc1fb5096fcfeeab969da762ceb83"
server_key_iv: "6e5283ada65f088f8eedd5e3e3270961"
server_key_version: "encryptionkey@0"
I20260929 20:51:19.489596 18048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:19.503752 18048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:19.504606 18048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:19.504875 18048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:19.522245 18048 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36067
I20260929 20:51:19.522264 18113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36067 every 8 connection(s)
I20260929 20:51:19.523528 18048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:51:19.526234 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 18048
I20260929 20:51:19.526386 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:51:19.526782 388 external_mini_cluster.cc:1428] Setting key 8e37ebd17abcd6d4c09343f05c06c1a9
I20260929 20:51:19.527258 18114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:19.541715 18114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0: Bootstrap starting.
I20260929 20:51:19.541908 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:19.528610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43142 (local address 127.0.97.62:36067)
0929 20:51:19.529648 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:19.529676 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:19.529725 (+ 49us) server_negotiation.cc:407] Connection header received
0929 20:51:19.531912 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:19.531943 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:19.532539 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:19.532994 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:19.533513 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:19.535472 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:19.536891 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:19.537899 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:19.538084 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:19.539200 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:19.539214 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:19.539236 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:19.539913 (+ 677us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:19.540004 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:19.540033 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:19.540135 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:19.540350 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 20:51:19.540652 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":344,"threads_started":1}
I20260929 20:51:19.544885 18114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:19.545775 18114 log.cc:824] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:19.548554 18114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0: No bootstrap required, opened a new log
I20260929 20:51:19.555546 18114 raft_consensus.cc:348] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c11d8edc0155470f833daf75986202e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36067 } }
I20260929 20:51:19.555862 18114 raft_consensus.cc:374] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:19.555930 18114 raft_consensus.cc:729] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c11d8edc0155470f833daf75986202e0, State: Initialized, Role: FOLLOWER
I20260929 20:51:19.556434 18114 consensus_queue.cc:260] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c11d8edc0155470f833daf75986202e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36067 } }
I20260929 20:51:19.556624 18114 raft_consensus.cc:388] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:51:19.556716 18114 raft_consensus.cc:482] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:51:19.556841 18114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:19.558844 18114 raft_consensus.cc:504] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c11d8edc0155470f833daf75986202e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36067 } }
I20260929 20:51:19.559433 18114 leader_election.cc:302] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c11d8edc0155470f833daf75986202e0; no voters:
I20260929 20:51:19.559944 18114 leader_election.cc:288] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:51:19.560076 18119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:19.560729 18119 raft_consensus.cc:686] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0 [term 1 LEADER]: Becoming Leader. State: Replica: c11d8edc0155470f833daf75986202e0, State: Running, Role: LEADER
I20260929 20:51:19.561216 18119 consensus_queue.cc:237] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c11d8edc0155470f833daf75986202e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36067 } }
I20260929 20:51:19.561831 18114 sys_catalog.cc:564] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:51:19.563680 18120 sys_catalog.cc:455] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c11d8edc0155470f833daf75986202e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c11d8edc0155470f833daf75986202e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36067 } } }
I20260929 20:51:19.565596 18121 sys_catalog.cc:455] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c11d8edc0155470f833daf75986202e0. Latest consensus state: current_term: 1 leader_uuid: "c11d8edc0155470f833daf75986202e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c11d8edc0155470f833daf75986202e0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36067 } } }
I20260929 20:51:19.565901 18121 sys_catalog.cc:458] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:19.565594 18120 sys_catalog.cc:458] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:19.567435 18129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:51:19.572872 18129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:51:19.581954 18129 catalog_manager.cc:1347] Generated new cluster ID: 854aa7fb653a4dabbde756e2393ecfcc
I20260929 20:51:19.582064 18129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:51:19.596169 18129 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:51:19.597450 18129 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:51:19.604586 18129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0: Generated new TSK 0
I20260929 20:51:19.605480 18129 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:51:19.621299 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36067
--builtin_ntp_servers=127.0.97.20:44463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:51:19.937539 18138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:19.937806 18138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:19.937917 18138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:19.945084 18138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:19.945179 18138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:19.945325 18138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:51:19.954233 18138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:19.955499 18138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:19.957165 18138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:19.969612 18143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:19.969683 18144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:19.969655 18146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:19.970512 18138 server_base.cc:1034] running on GCE node
I20260929 20:51:19.971093 18138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:19.971992 18138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:19.973228 18138 hybrid_clock.cc:648] HybridClock initialized: now 1790715079973171 us; error 44 us; skew 500 ppm
I20260929 20:51:19.973546 18138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:51:19.975864 18138 webserver.cc:460] Webserver started at http://127.0.97.1:35583/ using document root <none> and password file <none>
I20260929 20:51:19.976538 18138 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:19.976636 18138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:19.976948 18138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:19.979787 18138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "3b2848df38394af3af56bd014d24bafb"
format_stamp: "Formatted at 2026-09-29 20:51:19 on dist-test-slave-blv7"
server_key: "342d7df13126ed85f927c29b1fe19d4d"
server_key_iv: "9b7c04af9380393d7b7fb9d08fa071ad"
server_key_version: "encryptionkey@0"
I20260929 20:51:19.980527 18138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3b2848df38394af3af56bd014d24bafb"
format_stamp: "Formatted at 2026-09-29 20:51:19 on dist-test-slave-blv7"
server_key: "342d7df13126ed85f927c29b1fe19d4d"
server_key_iv: "9b7c04af9380393d7b7fb9d08fa071ad"
server_key_version: "encryptionkey@0"
I20260929 20:51:19.985523 18138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:51:19.988955 18152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:19.990269 18138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:51:19.990428 18138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "3b2848df38394af3af56bd014d24bafb"
format_stamp: "Formatted at 2026-09-29 20:51:19 on dist-test-slave-blv7"
server_key: "342d7df13126ed85f927c29b1fe19d4d"
server_key_iv: "9b7c04af9380393d7b7fb9d08fa071ad"
server_key_version: "encryptionkey@0"
I20260929 20:51:19.990625 18138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:20.013227 18138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:20.014037 18138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:20.014297 18138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:20.015266 18138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:51:20.016703 18138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:51:20.016793 18138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:20.016906 18138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:51:20.016973 18138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:20.047591 18138 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33653
I20260929 20:51:20.047657 18264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33653 every 8 connection(s)
I20260929 20:51:20.048834 18138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:51:20.056186 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 18138
I20260929 20:51:20.056347 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:51:20.056890 388 external_mini_cluster.cc:1428] Setting key 1e0757db1b0cc7afd30de8b135cbb767
I20260929 20:51:20.059705 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36067
--builtin_ntp_servers=127.0.97.20:44463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:51:20.061669 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:20.051340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54419 (local address 127.0.97.62:36067)
0929 20:51:20.051790 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:20.051796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:20.052909 (+ 1113us) server_negotiation.cc:407] Connection header received
0929 20:51:20.054598 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:20.054602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:20.054670 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:20.054765 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:20.055806 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:20.056916 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:20.058765 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:20.059385 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:20.059508 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:20.060864 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:20.060869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:20.060873 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:20.060893 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:20.060944 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:20.060958 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:20.061058 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:20.061230 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 20:51:20.061316 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":204,"threads_started":1}
I20260929 20:51:20.062572 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:20.051873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36067 (local address 127.0.97.1:54419)
0929 20:51:20.052767 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:20.052804 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:20.054352 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:20.054937 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:20.054944 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:20.055069 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:51:20.055656 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:20.055672 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:20.057092 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:20.057098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:20.058639 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:20.058650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:20.059590 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:20.059595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:20.059836 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:20.060557 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:51:20.060560 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:51:20.060596 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:51:20.060674 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:51:20.061062 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:51:20.061069 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:51:20.061074 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:51:20.061205 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 20:51:20.061515 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":257,"threads_started":1}
I20260929 20:51:20.064010 18265 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36067
I20260929 20:51:20.064330 18265 heartbeater.cc:464] Registering TS with master...
I20260929 20:51:20.065121 18265 heartbeater.cc:511] Master 127.0.97.62:36067 requested a full tablet report, sending...
I20260929 20:51:20.067423 18079 ts_manager.cc:194] Registered new tserver with Master: 3b2848df38394af3af56bd014d24bafb (127.0.97.1:33653)
I20260929 20:51:20.068876 18079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:54419
W20260929 20:51:20.371582 18269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:20.371877 18269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:20.372013 18269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:20.379046 18269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:20.379151 18269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:20.379261 18269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:51:20.388238 18269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:20.389474 18269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:20.391130 18269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:20.403632 18274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:20.403891 18275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:20.404155 18277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:20.404768 18269 server_base.cc:1034] running on GCE node
I20260929 20:51:20.405320 18269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:20.406214 18269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:20.407451 18269 hybrid_clock.cc:648] HybridClock initialized: now 1790715080407395 us; error 40 us; skew 500 ppm
I20260929 20:51:20.407764 18269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:51:20.409999 18269 webserver.cc:460] Webserver started at http://127.0.97.2:40469/ using document root <none> and password file <none>
I20260929 20:51:20.410656 18269 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:20.410751 18269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:20.411074 18269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:20.413836 18269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "2793cc694946444391245d5a18f2d0f6"
format_stamp: "Formatted at 2026-09-29 20:51:20 on dist-test-slave-blv7"
server_key: "ef6ad538c65fadc67b29222f3159e2e5"
server_key_iv: "a883ea4ee7fd2e6fa32bb170fb9af407"
server_key_version: "encryptionkey@0"
I20260929 20:51:20.414565 18269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2793cc694946444391245d5a18f2d0f6"
format_stamp: "Formatted at 2026-09-29 20:51:20 on dist-test-slave-blv7"
server_key: "ef6ad538c65fadc67b29222f3159e2e5"
server_key_iv: "a883ea4ee7fd2e6fa32bb170fb9af407"
server_key_version: "encryptionkey@0"
I20260929 20:51:20.419349 18269 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 20:51:20.422801 18283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:20.424166 18269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:51:20.424347 18269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "2793cc694946444391245d5a18f2d0f6"
format_stamp: "Formatted at 2026-09-29 20:51:20 on dist-test-slave-blv7"
server_key: "ef6ad538c65fadc67b29222f3159e2e5"
server_key_iv: "a883ea4ee7fd2e6fa32bb170fb9af407"
server_key_version: "encryptionkey@0"
I20260929 20:51:20.424539 18269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:20.454859 18269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:20.455741 18269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:20.455997 18269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:20.457000 18269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:51:20.458545 18269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:51:20.458637 18269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:20.458745 18269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:51:20.458811 18269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:20.489362 18269 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38441
I20260929 20:51:20.489425 18395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38441 every 8 connection(s)
I20260929 20:51:20.490581 18269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:51:20.494241 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 18269
I20260929 20:51:20.494414 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:51:20.494947 388 external_mini_cluster.cc:1428] Setting key c540ff12ec7587ec510308051b73c8cf
I20260929 20:51:20.497695 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36067
--builtin_ntp_servers=127.0.97.20:44463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:51:20.503371 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:20.493060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39665 (local address 127.0.97.62:36067)
0929 20:51:20.493243 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:20.493248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:20.494919 (+ 1671us) server_negotiation.cc:407] Connection header received
0929 20:51:20.497056 (+ 2137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:20.497061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:20.497141 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:20.497225 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:20.498398 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:20.499159 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:20.500716 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:20.501152 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:20.501262 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:20.502571 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:20.502575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:20.502579 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:20.502596 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:20.502626 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:20.502638 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:20.502711 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:20.502943 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 20:51:20.503066 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:51:20.504375 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:20.493614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36067 (local address 127.0.97.2:39665)
0929 20:51:20.494734 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:20.494774 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:20.496796 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:20.497498 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:20.497507 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:20.497597 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:51:20.498220 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:20.498238 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:20.499326 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:20.499330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:20.500569 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:20.500581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:20.501374 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:20.501382 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:20.501595 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:20.502242 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:51:20.502245 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:51:20.502295 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:51:20.502376 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:51:20.502731 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:51:20.502742 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:51:20.502750 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 20:51:20.502907 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 20:51:20.503337 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":357,"threads_started":1}
I20260929 20:51:20.505859 18396 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36067
I20260929 20:51:20.506155 18396 heartbeater.cc:464] Registering TS with master...
I20260929 20:51:20.506945 18396 heartbeater.cc:511] Master 127.0.97.62:36067 requested a full tablet report, sending...
I20260929 20:51:20.508431 18079 ts_manager.cc:194] Registered new tserver with Master: 2793cc694946444391245d5a18f2d0f6 (127.0.97.2:38441)
I20260929 20:51:20.509440 18079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:39665
W20260929 20:51:20.807636 18399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:20.807904 18399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:20.807986 18399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:20.815188 18399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:20.815285 18399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:20.815387 18399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:51:20.824503 18399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36067
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:20.825682 18399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:20.827416 18399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:20.840051 18405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:20.840087 18404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:20.840111 18407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:20.840843 18399 server_base.cc:1034] running on GCE node
I20260929 20:51:20.841482 18399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:20.842377 18399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:20.843580 18399 hybrid_clock.cc:648] HybridClock initialized: now 1790715080843530 us; error 33 us; skew 500 ppm
I20260929 20:51:20.843924 18399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:51:20.846210 18399 webserver.cc:460] Webserver started at http://127.0.97.3:35871/ using document root <none> and password file <none>
I20260929 20:51:20.846874 18399 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:20.846966 18399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:20.847298 18399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:20.850064 18399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "3dc05e02c8d34c73b8bc03b472f81141"
format_stamp: "Formatted at 2026-09-29 20:51:20 on dist-test-slave-blv7"
server_key: "aee7ab9e6111a4643bceb6651286c8b7"
server_key_iv: "e4aaf8d7fadeeb772b4c0fa8ed643570"
server_key_version: "encryptionkey@0"
I20260929 20:51:20.850798 18399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3dc05e02c8d34c73b8bc03b472f81141"
format_stamp: "Formatted at 2026-09-29 20:51:20 on dist-test-slave-blv7"
server_key: "aee7ab9e6111a4643bceb6651286c8b7"
server_key_iv: "e4aaf8d7fadeeb772b4c0fa8ed643570"
server_key_version: "encryptionkey@0"
I20260929 20:51:20.855655 18399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:51:20.859076 18413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:20.860450 18399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:51:20.860597 18399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "3dc05e02c8d34c73b8bc03b472f81141"
format_stamp: "Formatted at 2026-09-29 20:51:20 on dist-test-slave-blv7"
server_key: "aee7ab9e6111a4643bceb6651286c8b7"
server_key_iv: "e4aaf8d7fadeeb772b4c0fa8ed643570"
server_key_version: "encryptionkey@0"
I20260929 20:51:20.860795 18399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:20.885339 18399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:20.886117 18399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:20.886380 18399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:20.887384 18399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:51:20.888842 18399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:51:20.888934 18399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:20.889042 18399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:51:20.889113 18399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:20.919163 18399 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39655
I20260929 20:51:20.919255 18525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39655 every 8 connection(s)
I20260929 20:51:20.920397 18399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:51:20.920830 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 18399
I20260929 20:51:20.921010 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:51:20.921563 388 external_mini_cluster.cc:1428] Setting key 84cd81b44b3b8e4e11e49c4f38ace29d
I20260929 20:51:20.933056 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:20.923044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57497 (local address 127.0.97.62:36067)
0929 20:51:20.923211 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:20.923215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:20.924962 (+ 1747us) server_negotiation.cc:407] Connection header received
0929 20:51:20.926713 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:20.926718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:20.926797 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:20.926882 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:20.927973 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:20.928793 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:20.930293 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:20.930689 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:20.930797 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:20.932266 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:20.932271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:20.932275 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:20.932292 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:20.932322 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:20.932334 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:20.932408 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:20.932638 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 20:51:20.932733 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:51:20.934008 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:20.923721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36067 (local address 127.0.97.3:57497)
0929 20:51:20.924792 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:20.924864 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:20.926467 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:20.927152 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:20.927160 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:20.927254 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:51:20.927793 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:20.927819 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:20.928932 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:20.928936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:20.930157 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:20.930168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:20.930914 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:20.930922 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:20.931182 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:20.931930 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:51:20.931932 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:51:20.931970 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:51:20.932056 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:51:20.932431 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:51:20.932442 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:51:20.932450 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 20:51:20.932588 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 20:51:20.932975 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":353,"threads_started":1}
I20260929 20:51:20.935400 18526 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36067
I20260929 20:51:20.935721 18526 heartbeater.cc:464] Registering TS with master...
I20260929 20:51:20.936520 18526 heartbeater.cc:511] Master 127.0.97.62:36067 requested a full tablet report, sending...
I20260929 20:51:20.937824 18079 ts_manager.cc:194] Registered new tserver with Master: 3dc05e02c8d34c73b8bc03b472f81141 (127.0.97.3:39655)
I20260929 20:51:20.938622 18079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:57497
I20260929 20:51:20.949286 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:51:20.958938 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:20.953244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43156 (local address 127.0.97.62:36067)
0929 20:51:20.953465 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:20.953471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:20.953823 (+ 352us) server_negotiation.cc:407] Connection header received
0929 20:51:20.953971 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:20.953976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:20.954049 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:20.954136 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:20.954619 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:20.955380 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:20.956833 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:20.957198 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:20.957301 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:20.958276 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:20.958281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:20.958284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:20.958302 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:20.958336 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:20.958347 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:20.958421 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:20.958567 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 20:51:20.958653 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:51:20.962579 18079 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:51:20.969523 18079 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:51:21.001798 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:20.982577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33653 (local address 127.0.0.1:42124)
0929 20:51:20.993567 (+ 10990us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:20.993585 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:20.993787 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:20.994646 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:20.994650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:20.994674 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:20.995177 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:20.995194 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:20.997046 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:20.997051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:20.999170 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:20.999185 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:21.000692 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:21.000699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:21.000996 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:21.001057 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 20:51:21.001176 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 20:51:21.001379 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10877,"mutex_wait_us":27,"thread_start_us":10750,"threads_started":1}
I20260929 20:51:21.001821 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:20.982318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58950 (local address 127.0.97.3:39655)
0929 20:51:20.994161 (+ 11843us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:20.994169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:20.994198 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:51:20.994294 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:20.994301 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:20.994558 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:20.995515 (+ 957us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:20.995802 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:20.997193 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:20.999135 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:21.000538 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:21.000944 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:21.001214 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:21.001300 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 20:51:21.001470 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11483,"thread_start_us":11284,"threads_started":1}
I20260929 20:51:21.002147 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:20.981594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37946 (local address 127.0.97.2:38441)
0929 20:51:20.993285 (+ 11691us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:20.993304 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:20.993338 (+ 34us) server_negotiation.cc:407] Connection header received
0929 20:51:20.993552 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:20.993561 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:20.993890 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:20.994918 (+ 1028us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:20.995219 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:20.996276 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:20.999387 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:21.000686 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:21.001176 (+ 490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:21.001415 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:21.001477 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 20:51:21.001618 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11509,"thread_start_us":11378,"threads_started":1}
I20260929 20:51:21.002105 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:20.981732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39655 (local address 127.0.0.1:58950)
0929 20:51:20.992625 (+ 10893us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:20.992647 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:20.992777 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:20.994774 (+ 1997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:20.994782 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:20.994807 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:20.995313 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:20.995328 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:20.997325 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:20.997332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:20.998883 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:20.998897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:21.000756 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:21.000763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:21.001073 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:21.001148 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 20:51:21.001208 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 20:51:21.001823 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10710,"mutex_wait_us":622,"thread_start_us":10373,"threads_started":1}
I20260929 20:51:21.002662 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:20.983127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42124 (local address 127.0.97.1:33653)
0929 20:51:20.990646 (+ 7519us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:20.990656 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:20.993687 (+ 3031us) server_negotiation.cc:407] Connection header received
0929 20:51:20.993941 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:20.993949 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:20.994179 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:20.994411 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:20.995361 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:20.996886 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:20.999390 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:21.000492 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:21.000614 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:21.000864 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:21.002183 (+ 1319us) server_negotiation.cc:299] Negotiation successful
0929 20:51:21.002348 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7351,"thread_start_us":7627,"threads_started":1}
I20260929 20:51:21.002210 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:20.981305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38441 (local address 127.0.0.1:37946)
0929 20:51:20.982129 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:20.982170 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:20.982366 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:20.994175 (+ 11809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:20.994182 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:20.994224 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:20.994640 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:20.994653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:20.996435 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:20.996438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:20.998674 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:20.998689 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:21.000893 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:21.000899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:21.001116 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:21.001285 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 20:51:21.001340 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 20:51:21.001441 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":163,"thread_start_us":399,"threads_started":1}
I20260929 20:51:21.006106 18331 tablet_service.cc:1467] Processing CreateTablet for tablet d5aca13a090d4de6b54b42f4c5f98758 (DEFAULT_TABLE table=test-table [id=220c306ab59a417184a24219cdb3a170]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:51:21.006244 18461 tablet_service.cc:1467] Processing CreateTablet for tablet d5aca13a090d4de6b54b42f4c5f98758 (DEFAULT_TABLE table=test-table [id=220c306ab59a417184a24219cdb3a170]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:51:21.006244 18200 tablet_service.cc:1467] Processing CreateTablet for tablet d5aca13a090d4de6b54b42f4c5f98758 (DEFAULT_TABLE table=test-table [id=220c306ab59a417184a24219cdb3a170]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:51:21.007375 18331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5aca13a090d4de6b54b42f4c5f98758. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:21.007596 18200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5aca13a090d4de6b54b42f4c5f98758. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:21.007596 18461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5aca13a090d4de6b54b42f4c5f98758. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:21.018785 18544 tablet_bootstrap.cc:492] T d5aca13a090d4de6b54b42f4c5f98758 P 2793cc694946444391245d5a18f2d0f6: Bootstrap starting.
I20260929 20:51:21.021311 18545 tablet_bootstrap.cc:492] T d5aca13a090d4de6b54b42f4c5f98758 P 3dc05e02c8d34c73b8bc03b472f81141: Bootstrap starting.
I20260929 20:51:21.022322 18544 tablet_bootstrap.cc:654] T d5aca13a090d4de6b54b42f4c5f98758 P 2793cc694946444391245d5a18f2d0f6: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:21.022970 18546 tablet_bootstrap.cc:492] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb: Bootstrap starting.
I20260929 20:51:21.023744 18544 log.cc:824] T d5aca13a090d4de6b54b42f4c5f98758 P 2793cc694946444391245d5a18f2d0f6: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:21.024675 18545 tablet_bootstrap.cc:654] T d5aca13a090d4de6b54b42f4c5f98758 P 3dc05e02c8d34c73b8bc03b472f81141: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:21.025804 18545 log.cc:824] T d5aca13a090d4de6b54b42f4c5f98758 P 3dc05e02c8d34c73b8bc03b472f81141: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:21.025992 18546 tablet_bootstrap.cc:654] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:21.027108 18544 tablet_bootstrap.cc:492] T d5aca13a090d4de6b54b42f4c5f98758 P 2793cc694946444391245d5a18f2d0f6: No bootstrap required, opened a new log
I20260929 20:51:21.027151 18546 log.cc:824] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:21.027403 18544 ts_tablet_manager.cc:1397] T d5aca13a090d4de6b54b42f4c5f98758 P 2793cc694946444391245d5a18f2d0f6: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 20:51:21.030010 18545 tablet_bootstrap.cc:492] T d5aca13a090d4de6b54b42f4c5f98758 P 3dc05e02c8d34c73b8bc03b472f81141: No bootstrap required, opened a new log
I20260929 20:51:21.030165 18546 tablet_bootstrap.cc:492] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb: No bootstrap required, opened a new log
I20260929 20:51:21.030280 18545 ts_tablet_manager.cc:1397] T d5aca13a090d4de6b54b42f4c5f98758 P 3dc05e02c8d34c73b8bc03b472f81141: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 20:51:21.030411 18546 ts_tablet_manager.cc:1397] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 20:51:21.037923 18544 raft_consensus.cc:348] T d5aca13a090d4de6b54b42f4c5f98758 P 2793cc694946444391245d5a18f2d0f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2793cc694946444391245d5a18f2d0f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38441 } } peers { permanent_uuid: "3b2848df38394af3af56bd014d24bafb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33653 } } peers { permanent_uuid: "3dc05e02c8d34c73b8bc03b472f81141" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39655 } }
I20260929 20:51:21.038175 18545 raft_consensus.cc:348] T d5aca13a090d4de6b54b42f4c5f98758 P 3dc05e02c8d34c73b8bc03b472f81141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2793cc694946444391245d5a18f2d0f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38441 } } peers { permanent_uuid: "3b2848df38394af3af56bd014d24bafb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33653 } } peers { permanent_uuid: "3dc05e02c8d34c73b8bc03b472f81141" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39655 } }
I20260929 20:51:21.038430 18544 raft_consensus.cc:374] T d5aca13a090d4de6b54b42f4c5f98758 P 2793cc694946444391245d5a18f2d0f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:21.038529 18544 raft_consensus.cc:729] T d5aca13a090d4de6b54b42f4c5f98758 P 2793cc694946444391245d5a18f2d0f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2793cc694946444391245d5a18f2d0f6, State: Initialized, Role: FOLLOWER
I20260929 20:51:21.038560 18545 raft_consensus.cc:374] T d5aca13a090d4de6b54b42f4c5f98758 P 3dc05e02c8d34c73b8bc03b472f81141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:21.038650 18545 raft_consensus.cc:729] T d5aca13a090d4de6b54b42f4c5f98758 P 3dc05e02c8d34c73b8bc03b472f81141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc05e02c8d34c73b8bc03b472f81141, State: Initialized, Role: FOLLOWER
I20260929 20:51:21.039324 18545 consensus_queue.cc:260] T d5aca13a090d4de6b54b42f4c5f98758 P 3dc05e02c8d34c73b8bc03b472f81141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2793cc694946444391245d5a18f2d0f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38441 } } peers { permanent_uuid: "3b2848df38394af3af56bd014d24bafb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33653 } } peers { permanent_uuid: "3dc05e02c8d34c73b8bc03b472f81141" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39655 } }
I20260929 20:51:21.039322 18544 consensus_queue.cc:260] T d5aca13a090d4de6b54b42f4c5f98758 P 2793cc694946444391245d5a18f2d0f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2793cc694946444391245d5a18f2d0f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38441 } } peers { permanent_uuid: "3b2848df38394af3af56bd014d24bafb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33653 } } peers { permanent_uuid: "3dc05e02c8d34c73b8bc03b472f81141" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39655 } }
I20260929 20:51:21.040421 18526 heartbeater.cc:503] Master 127.0.97.62:36067 was elected leader, sending a full tablet report...
I20260929 20:51:21.040922 18545 ts_tablet_manager.cc:1428] T d5aca13a090d4de6b54b42f4c5f98758 P 3dc05e02c8d34c73b8bc03b472f81141: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260929 20:51:21.041410 18546 raft_consensus.cc:348] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2793cc694946444391245d5a18f2d0f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38441 } } peers { permanent_uuid: "3b2848df38394af3af56bd014d24bafb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33653 } } peers { permanent_uuid: "3dc05e02c8d34c73b8bc03b472f81141" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39655 } }
I20260929 20:51:21.041746 18546 raft_consensus.cc:374] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:21.041843 18546 raft_consensus.cc:729] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b2848df38394af3af56bd014d24bafb, State: Initialized, Role: FOLLOWER
I20260929 20:51:21.042433 18546 consensus_queue.cc:260] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2793cc694946444391245d5a18f2d0f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38441 } } peers { permanent_uuid: "3b2848df38394af3af56bd014d24bafb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33653 } } peers { permanent_uuid: "3dc05e02c8d34c73b8bc03b472f81141" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39655 } }
I20260929 20:51:21.042604 18396 heartbeater.cc:503] Master 127.0.97.62:36067 was elected leader, sending a full tablet report...
I20260929 20:51:21.043144 18544 ts_tablet_manager.cc:1428] T d5aca13a090d4de6b54b42f4c5f98758 P 2793cc694946444391245d5a18f2d0f6: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260929 20:51:21.044008 18265 heartbeater.cc:503] Master 127.0.97.62:36067 was elected leader, sending a full tablet report...
I20260929 20:51:21.044605 18546 ts_tablet_manager.cc:1428] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260929 20:51:21.050863 18266 tablet.cc:2369] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:51:21.091718 18552 raft_consensus.cc:482] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:51:21.091991 18552 raft_consensus.cc:504] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2793cc694946444391245d5a18f2d0f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38441 } } peers { permanent_uuid: "3b2848df38394af3af56bd014d24bafb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33653 } } peers { permanent_uuid: "3dc05e02c8d34c73b8bc03b472f81141" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39655 } }
I20260929 20:51:21.093801 18552 leader_election.cc:288] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2793cc694946444391245d5a18f2d0f6 (127.0.97.2:38441), 3dc05e02c8d34c73b8bc03b472f81141 (127.0.97.3:39655)
I20260929 20:51:21.102632 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:21.094355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39249 (local address 127.0.97.2:38441)
0929 20:51:21.094779 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:21.094786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:21.095250 (+ 464us) server_negotiation.cc:407] Connection header received
0929 20:51:21.095358 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:21.095363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:21.095431 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:21.095509 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:21.096212 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:21.097302 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:21.100347 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:21.101578 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:21.101715 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:21.101844 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:21.102245 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 20:51:21.102381 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288}
I20260929 20:51:21.102638 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:21.094341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34741 (local address 127.0.97.3:39655)
0929 20:51:21.094615 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:21.094623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:21.094996 (+ 373us) server_negotiation.cc:407] Connection header received
0929 20:51:21.095249 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:21.095255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:21.095333 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:21.095429 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:21.096147 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:21.097302 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:21.100375 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:21.101605 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:21.101715 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:21.101816 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:21.102245 (+ 429us) server_negotiation.cc:299] Negotiation successful
0929 20:51:21.102422 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 20:51:21.102733 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:21.094173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38441 (local address 127.0.97.1:39249)
0929 20:51:21.095153 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:21.095168 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:21.095266 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:21.095533 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:21.095537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:21.095559 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:21.096003 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:21.096012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:21.097462 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:21.097468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:21.100110 (+ 2642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:21.100125 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:21.101823 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:21.101829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:21.102009 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:21.102078 (+ 69us) client_negotiation.cc:769] Sending connection context
0929 20:51:21.102166 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 20:51:21.102324 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":348,"thread_start_us":683,"threads_started":1}
I20260929 20:51:21.103184 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:21.094111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39655 (local address 127.0.97.1:34741)
0929 20:51:21.094872 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:21.094892 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:21.095092 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:21.095459 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:21.095463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:21.095487 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:21.095961 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:21.095977 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:21.097463 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:21.097468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:21.100112 (+ 2644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:21.100125 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:21.101802 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:21.101810 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:21.102008 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:21.102075 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 20:51:21.102166 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 20:51:21.102350 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":190,"spinlock_wait_cycles":3456,"thread_start_us":444,"threads_started":1}
I20260929 20:51:21.103791 18351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5aca13a090d4de6b54b42f4c5f98758" candidate_uuid: "3b2848df38394af3af56bd014d24bafb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2793cc694946444391245d5a18f2d0f6" is_pre_election: true
I20260929 20:51:21.104128 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5aca13a090d4de6b54b42f4c5f98758" candidate_uuid: "3b2848df38394af3af56bd014d24bafb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc05e02c8d34c73b8bc03b472f81141" is_pre_election: true
I20260929 20:51:21.104265 18351 raft_consensus.cc:2454] T d5aca13a090d4de6b54b42f4c5f98758 P 2793cc694946444391245d5a18f2d0f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b2848df38394af3af56bd014d24bafb in term 0.
I20260929 20:51:21.104596 18481 raft_consensus.cc:2454] T d5aca13a090d4de6b54b42f4c5f98758 P 3dc05e02c8d34c73b8bc03b472f81141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b2848df38394af3af56bd014d24bafb in term 0.
I20260929 20:51:21.104897 18154 leader_election.cc:302] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2793cc694946444391245d5a18f2d0f6, 3b2848df38394af3af56bd014d24bafb; no voters:
I20260929 20:51:21.105305 18552 raft_consensus.cc:2781] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:51:21.105451 18552 raft_consensus.cc:482] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:51:21.105551 18552 raft_consensus.cc:3037] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:21.107673 18552 raft_consensus.cc:504] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2793cc694946444391245d5a18f2d0f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38441 } } peers { permanent_uuid: "3b2848df38394af3af56bd014d24bafb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33653 } } peers { permanent_uuid: "3dc05e02c8d34c73b8bc03b472f81141" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39655 } }
I20260929 20:51:21.108247 18552 leader_election.cc:288] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb [CANDIDATE]: Term 1 election: Requested vote from peers 2793cc694946444391245d5a18f2d0f6 (127.0.97.2:38441), 3dc05e02c8d34c73b8bc03b472f81141 (127.0.97.3:39655)
I20260929 20:51:21.108672 18351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5aca13a090d4de6b54b42f4c5f98758" candidate_uuid: "3b2848df38394af3af56bd014d24bafb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2793cc694946444391245d5a18f2d0f6"
I20260929 20:51:21.108737 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5aca13a090d4de6b54b42f4c5f98758" candidate_uuid: "3b2848df38394af3af56bd014d24bafb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc05e02c8d34c73b8bc03b472f81141"
I20260929 20:51:21.108856 18351 raft_consensus.cc:3037] T d5aca13a090d4de6b54b42f4c5f98758 P 2793cc694946444391245d5a18f2d0f6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:21.108901 18481 raft_consensus.cc:3037] T d5aca13a090d4de6b54b42f4c5f98758 P 3dc05e02c8d34c73b8bc03b472f81141 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:21.111316 18481 raft_consensus.cc:2454] T d5aca13a090d4de6b54b42f4c5f98758 P 3dc05e02c8d34c73b8bc03b472f81141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b2848df38394af3af56bd014d24bafb in term 1.
I20260929 20:51:21.111398 18351 raft_consensus.cc:2454] T d5aca13a090d4de6b54b42f4c5f98758 P 2793cc694946444391245d5a18f2d0f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b2848df38394af3af56bd014d24bafb in term 1.
I20260929 20:51:21.111690 18156 leader_election.cc:302] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b2848df38394af3af56bd014d24bafb, 3dc05e02c8d34c73b8bc03b472f81141; no voters:
I20260929 20:51:21.111950 18552 raft_consensus.cc:2781] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:21.112399 18552 raft_consensus.cc:686] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb [term 1 LEADER]: Becoming Leader. State: Replica: 3b2848df38394af3af56bd014d24bafb, State: Running, Role: LEADER
I20260929 20:51:21.112825 18552 consensus_queue.cc:237] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2793cc694946444391245d5a18f2d0f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38441 } } peers { permanent_uuid: "3b2848df38394af3af56bd014d24bafb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33653 } } peers { permanent_uuid: "3dc05e02c8d34c73b8bc03b472f81141" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39655 } }
I20260929 20:51:21.117210 18078 catalog_manager.cc:5495] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b2848df38394af3af56bd014d24bafb (127.0.97.1). New cstate: current_term: 1 leader_uuid: "3b2848df38394af3af56bd014d24bafb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2793cc694946444391245d5a18f2d0f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b2848df38394af3af56bd014d24bafb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dc05e02c8d34c73b8bc03b472f81141" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39655 } health_report { overall_health: UNKNOWN } } }
I20260929 20:51:21.152149 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:21.146943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42138 (local address 127.0.97.1:33653)
0929 20:51:21.147147 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:21.147152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:21.147169 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:51:21.147258 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:21.147262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:21.147356 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:21.147478 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:51:21.147820 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:21.148505 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:21.149821 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:21.150269 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:21.150364 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:21.150883 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:51:21.151454 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:51:21.151568 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:21.151675 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 20:51:21.151881 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:51:21.160176 18481 raft_consensus.cc:1261] T d5aca13a090d4de6b54b42f4c5f98758 P 3dc05e02c8d34c73b8bc03b472f81141 [term 1 FOLLOWER]: Refusing update from remote peer 3b2848df38394af3af56bd014d24bafb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:51:21.160176 18351 raft_consensus.cc:1261] T d5aca13a090d4de6b54b42f4c5f98758 P 2793cc694946444391245d5a18f2d0f6 [term 1 FOLLOWER]: Refusing update from remote peer 3b2848df38394af3af56bd014d24bafb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:51:21.161196 18555 consensus_queue.cc:1035] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc05e02c8d34c73b8bc03b472f81141" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:51:21.161571 18552 consensus_queue.cc:1035] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2793cc694946444391245d5a18f2d0f6" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 20:51:21.172166 18527 tablet.cc:2369] T d5aca13a090d4de6b54b42f4c5f98758 P 3dc05e02c8d34c73b8bc03b472f81141: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:51:21.176312 18558 mvcc.cc:205] Tried to move back new op lower bound from 7334768972422639616 to 7334768972241604608. Current Snapshot: MvccSnapshot[applied={T|T < 7334768972422639616}]
I20260929 20:51:21.189141 18078 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:51:21.189380 18078 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:51:21.192762 18078 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c11d8edc0155470f833daf75986202e0: Sending DeleteTablet for 3 replicas of tablet d5aca13a090d4de6b54b42f4c5f98758
I20260929 20:51:21.193922 18200 tablet_service.cc:1514] Processing DeleteTablet for tablet d5aca13a090d4de6b54b42f4c5f98758 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:51:21 UTC) from {username='slave'} at 127.0.0.1:42124
I20260929 20:51:21.193895 18331 tablet_service.cc:1514] Processing DeleteTablet for tablet d5aca13a090d4de6b54b42f4c5f98758 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:51:21 UTC) from {username='slave'} at 127.0.0.1:37946
I20260929 20:51:21.194514 18461 tablet_service.cc:1514] Processing DeleteTablet for tablet d5aca13a090d4de6b54b42f4c5f98758 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:51:21 UTC) from {username='slave'} at 127.0.0.1:58950
I20260929 20:51:21.194981 18567 tablet_replica.cc:330] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb: stopping tablet replica
I20260929 20:51:21.195175 18568 tablet_replica.cc:330] T d5aca13a090d4de6b54b42f4c5f98758 P 2793cc694946444391245d5a18f2d0f6: stopping tablet replica
I20260929 20:51:21.195501 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 18138
I20260929 20:51:21.195577 18568 raft_consensus.cc:2229] T d5aca13a090d4de6b54b42f4c5f98758 P 2793cc694946444391245d5a18f2d0f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:51:21.195643 18567 raft_consensus.cc:2229] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:51:21.196044 18568 raft_consensus.cc:2258] T d5aca13a090d4de6b54b42f4c5f98758 P 2793cc694946444391245d5a18f2d0f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:51:21.196437 18567 raft_consensus.cc:2258] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:51:21.198184 18568 ts_tablet_manager.cc:1905] T d5aca13a090d4de6b54b42f4c5f98758 P 2793cc694946444391245d5a18f2d0f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:51:21.205294 18568 ts_tablet_manager.cc:1918] T d5aca13a090d4de6b54b42f4c5f98758 P 2793cc694946444391245d5a18f2d0f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:51:21.205443 18568 log.cc:1196] T d5aca13a090d4de6b54b42f4c5f98758 P 2793cc694946444391245d5a18f2d0f6: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/d5aca13a090d4de6b54b42f4c5f98758
I20260929 20:51:21.205590 18567 ts_tablet_manager.cc:1905] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:51:21.205933 18568 ts_tablet_manager.cc:1939] T d5aca13a090d4de6b54b42f4c5f98758 P 2793cc694946444391245d5a18f2d0f6: Deleting consensus metadata
I20260929 20:51:21.207674 18064 catalog_manager.cc:4841] TS 2793cc694946444391245d5a18f2d0f6 (127.0.97.2:38441): tablet d5aca13a090d4de6b54b42f4c5f98758 (table test-table [id=220c306ab59a417184a24219cdb3a170]) successfully deleted
I20260929 20:51:21.209014 18569 tablet_replica.cc:330] T d5aca13a090d4de6b54b42f4c5f98758 P 3dc05e02c8d34c73b8bc03b472f81141: stopping tablet replica
I20260929 20:51:21.209429 18569 raft_consensus.cc:2229] T d5aca13a090d4de6b54b42f4c5f98758 P 3dc05e02c8d34c73b8bc03b472f81141 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:51:21.209892 18569 raft_consensus.cc:2258] T d5aca13a090d4de6b54b42f4c5f98758 P 3dc05e02c8d34c73b8bc03b472f81141 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:51:21.211973 18569 ts_tablet_manager.cc:1905] T d5aca13a090d4de6b54b42f4c5f98758 P 3dc05e02c8d34c73b8bc03b472f81141: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:51:21.214335 18567 ts_tablet_manager.cc:1918] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:51:21.214491 18567 log.cc:1196] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/d5aca13a090d4de6b54b42f4c5f98758
I20260929 20:51:21.215008 18567 ts_tablet_manager.cc:1939] T d5aca13a090d4de6b54b42f4c5f98758 P 3b2848df38394af3af56bd014d24bafb: Deleting consensus metadata
I20260929 20:51:21.216581 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:21.197791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.0.97.1:33653)
0929 20:51:21.197953 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:21.197959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:21.208361 (+ 10402us) server_negotiation.cc:407] Connection header received
0929 20:51:21.208547 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:21.208552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:21.208667 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:21.208771 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:21.209317 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:21.210545 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:21.212456 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:21.213171 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:21.213273 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:21.214715 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:21.214726 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:21.214743 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:21.215622 (+ 879us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:21.215684 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:21.215708 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:21.215832 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:21.216094 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 20:51:21.216205 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:51:21.216681 18064 catalog_manager.cc:4841] TS 3b2848df38394af3af56bd014d24bafb (127.0.97.1:33653): tablet d5aca13a090d4de6b54b42f4c5f98758 (table test-table [id=220c306ab59a417184a24219cdb3a170]) successfully deleted
I20260929 20:51:21.217650 18260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:21.218406 18569 ts_tablet_manager.cc:1918] T d5aca13a090d4de6b54b42f4c5f98758 P 3dc05e02c8d34c73b8bc03b472f81141: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:51:21.218547 18569 log.cc:1196] T d5aca13a090d4de6b54b42f4c5f98758 P 3dc05e02c8d34c73b8bc03b472f81141: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/d5aca13a090d4de6b54b42f4c5f98758
I20260929 20:51:21.219002 18569 ts_tablet_manager.cc:1939] T d5aca13a090d4de6b54b42f4c5f98758 P 3dc05e02c8d34c73b8bc03b472f81141: Deleting consensus metadata
I20260929 20:51:21.220541 18066 catalog_manager.cc:4841] TS 3dc05e02c8d34c73b8bc03b472f81141 (127.0.97.3:39655): tablet d5aca13a090d4de6b54b42f4c5f98758 (table test-table [id=220c306ab59a417184a24219cdb3a170]) successfully deleted
I20260929 20:51:21.312119 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 18138
I20260929 20:51:21.327189 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 18269
I20260929 20:51:21.333707 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:21.328023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37956 (local address 127.0.97.2:38441)
0929 20:51:21.328191 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:21.328196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:21.328214 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:51:21.328286 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:21.328290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:21.328384 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:21.328490 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:21.328838 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:21.329590 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:21.330938 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:21.331397 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:21.331498 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:21.332476 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:21.332484 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:21.332501 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:21.333050 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:21.333100 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:21.333119 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:21.333200 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:21.333299 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 20:51:21.333382 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:51:21.334421 18391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:21.419132 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 18269
I20260929 20:51:21.432579 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 18399
I20260929 20:51:21.439167 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:21.433364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58960 (local address 127.0.97.3:39655)
0929 20:51:21.433533 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:21.433537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:21.433555 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:51:21.433630 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:21.433634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:21.433733 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:21.433812 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:21.434129 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:21.434929 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:21.436354 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:21.436792 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:21.436894 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:21.437818 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:21.437827 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:21.437843 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:21.438435 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:21.438483 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:21.438510 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:21.438600 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:21.438704 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 20:51:21.438787 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:51:21.439801 18521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:21.526888 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 18399
I20260929 20:51:21.546168 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 18048
I20260929 20:51:21.547232 18109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:21.652294 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 18048
2026-09-29T20:51:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2606 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:51:21 dist-test-slave-blv7 krb5kdc[18577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:51:21 dist-test-slave-blv7 krb5kdc[18577](info): set up 2 sockets
Sep 29 20:51:21 dist-test-slave-blv7 krb5kdc[18577](info): commencing operation
krb5kdc: starting...
W20260929 20:51:22.678828 388 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:51:22 dist-test-slave-blv7 krb5kdc[18577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:51:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:51:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:51:22.906160 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45681
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:23.219048 18590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:23.219321 18590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:23.219417 18590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:23.226455 18590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:51:23.226557 18590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:23.226600 18590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:23.226650 18590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:51:23.226691 18590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:51:23.236366 18590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:23.237593 18590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:23.238834 18590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:23.250422 18598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:23.250458 18596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:23.250437 18595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:23.251546 18590 server_base.cc:1034] running on GCE node
I20260929 20:51:23.252345 18590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:23.253877 18590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:23.255146 18590 hybrid_clock.cc:648] HybridClock initialized: now 1790715083255092 us; error 32 us; skew 500 ppm
I20260929 20:51:23.255502 18590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:23 dist-test-slave-blv7 krb5kdc[18577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:23.258579 18590 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:51:23.260186 18590 webserver.cc:460] Webserver started at http://127.0.97.62:45321/ using document root <none> and password file <none>
I20260929 20:51:23.260865 18590 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:23.260991 18590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:23.261312 18590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:23.264395 18590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "041ea2a5cddb4066adad68932cdf1bf8"
format_stamp: "Formatted at 2026-09-29 20:51:23 on dist-test-slave-blv7"
server_key: "4a0b8880a2f5b613c9e30fd49fe3aafe"
server_key_iv: "c7e99ad45ed0b496a4f8103021f782f3"
server_key_version: "encryptionkey@0"
I20260929 20:51:23.265120 18590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "041ea2a5cddb4066adad68932cdf1bf8"
format_stamp: "Formatted at 2026-09-29 20:51:23 on dist-test-slave-blv7"
server_key: "4a0b8880a2f5b613c9e30fd49fe3aafe"
server_key_iv: "c7e99ad45ed0b496a4f8103021f782f3"
server_key_version: "encryptionkey@0"
I20260929 20:51:23.270291 18590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:51:23.273741 18605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:23.275053 18590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:51:23.275206 18590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "041ea2a5cddb4066adad68932cdf1bf8"
format_stamp: "Formatted at 2026-09-29 20:51:23 on dist-test-slave-blv7"
server_key: "4a0b8880a2f5b613c9e30fd49fe3aafe"
server_key_iv: "c7e99ad45ed0b496a4f8103021f782f3"
server_key_version: "encryptionkey@0"
I20260929 20:51:23.275390 18590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:23.296059 18590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:23.300558 18590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:23.300829 18590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:23.318136 18590 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45681
I20260929 20:51:23.318161 18656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45681 every 8 connection(s)
I20260929 20:51:23.319558 18590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:51:23.323295 18657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:23.327742 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 18590
I20260929 20:51:23.327922 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:51:23.328444 388 external_mini_cluster.cc:1428] Setting key 6021a2aa88df9c39e3c925feb5c980d4
I20260929 20:51:23.334406 18657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8: Bootstrap starting.
I20260929 20:51:23.337834 18657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:23.339051 18657 log.cc:824] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:23.341838 18657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8: No bootstrap required, opened a new log
Sep 29 20:51:23 dist-test-slave-blv7 krb5kdc[18577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:51:23.351688 18657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041ea2a5cddb4066adad68932cdf1bf8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45681 } }
I20260929 20:51:23.352140 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:23.330330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51078 (local address 127.0.97.62:45681)
0929 20:51:23.331208 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:23.331224 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:23.331274 (+ 50us) server_negotiation.cc:407] Connection header received
0929 20:51:23.333174 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:23.333198 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:23.333580 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:23.333915 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:23.335639 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:23.337040 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:23.338806 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:23.339644 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:23.339866 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:23.343659 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:23.343669 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:23.343688 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:23.343740 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:23.347595 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:23.348405 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:23.348413 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:23.348422 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:23.348628 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:23.349027 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:23.349032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:23.349036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:23.349775 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:23.350077 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:23.350363 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 20:51:23.350818 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":398,"threads_started":1}
I20260929 20:51:23.352228 18657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:23.352336 18657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 041ea2a5cddb4066adad68932cdf1bf8, State: Initialized, Role: FOLLOWER
I20260929 20:51:23.352882 18657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041ea2a5cddb4066adad68932cdf1bf8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45681 } }
I20260929 20:51:23.353092 18657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:51:23.353219 18657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:51:23.353374 18657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:23.355683 18657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041ea2a5cddb4066adad68932cdf1bf8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45681 } }
I20260929 20:51:23.356148 18657 leader_election.cc:302] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 041ea2a5cddb4066adad68932cdf1bf8; no voters:
I20260929 20:51:23.356720 18657 leader_election.cc:288] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:51:23.356870 18662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:23.357532 18662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8 [term 1 LEADER]: Becoming Leader. State: Replica: 041ea2a5cddb4066adad68932cdf1bf8, State: Running, Role: LEADER
I20260929 20:51:23.358044 18662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041ea2a5cddb4066adad68932cdf1bf8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45681 } }
I20260929 20:51:23.358690 18657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:51:23.360745 18664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 041ea2a5cddb4066adad68932cdf1bf8. Latest consensus state: current_term: 1 leader_uuid: "041ea2a5cddb4066adad68932cdf1bf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041ea2a5cddb4066adad68932cdf1bf8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45681 } } }
I20260929 20:51:23.361155 18664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:23.361541 18663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "041ea2a5cddb4066adad68932cdf1bf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "041ea2a5cddb4066adad68932cdf1bf8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45681 } } }
I20260929 20:51:23.361887 18663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:23.365898 18671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:51:23.370760 18671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:51:23.379910 18671 catalog_manager.cc:1347] Generated new cluster ID: b1a431ddc0c946c9b6667a7810791384
I20260929 20:51:23.380038 18671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:51:23.395375 18671 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:51:23.396777 18671 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:51:23.403677 18671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8: Generated new TSK 0
I20260929 20:51:23.404492 18671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:51:23.471781 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45681
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:23.789816 18683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:23.790082 18683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:23.790184 18683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:23.797360 18683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:23.797467 18683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:23.797577 18683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:51:23.807235 18683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:23.808454 18683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:23.809686 18683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:23.822096 18691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:23.822057 18688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:23.822117 18689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:23.822939 18683 server_base.cc:1034] running on GCE node
I20260929 20:51:23.823553 18683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:23.824507 18683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:23.825779 18683 hybrid_clock.cc:648] HybridClock initialized: now 1790715083825706 us; error 57 us; skew 500 ppm
I20260929 20:51:23.826117 18683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:23 dist-test-slave-blv7 krb5kdc[18577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:23.829284 18683 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:51:23.830690 18683 webserver.cc:460] Webserver started at http://127.0.97.1:35473/ using document root <none> and password file <none>
I20260929 20:51:23.831405 18683 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:23.831521 18683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:23.831825 18683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:23.834759 18683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "0b365d7d5d4a40ac8ccbbe1b6610f326"
format_stamp: "Formatted at 2026-09-29 20:51:23 on dist-test-slave-blv7"
server_key: "f33022a461147cc0140cadb09f34d507"
server_key_iv: "5dc6341f971a5740be76632d67af028e"
server_key_version: "encryptionkey@0"
I20260929 20:51:23.835539 18683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0b365d7d5d4a40ac8ccbbe1b6610f326"
format_stamp: "Formatted at 2026-09-29 20:51:23 on dist-test-slave-blv7"
server_key: "f33022a461147cc0140cadb09f34d507"
server_key_iv: "5dc6341f971a5740be76632d67af028e"
server_key_version: "encryptionkey@0"
I20260929 20:51:23.840468 18683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 20:51:23.843799 18698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:23.845058 18683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:51:23.845211 18683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "0b365d7d5d4a40ac8ccbbe1b6610f326"
format_stamp: "Formatted at 2026-09-29 20:51:23 on dist-test-slave-blv7"
server_key: "f33022a461147cc0140cadb09f34d507"
server_key_iv: "5dc6341f971a5740be76632d67af028e"
server_key_version: "encryptionkey@0"
I20260929 20:51:23.845394 18683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:23.862054 18683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:23.866073 18683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:23.866325 18683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:23.867252 18683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:51:23.868480 18683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:51:23.868553 18683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:23.868677 18683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:51:23.868737 18683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:23.898350 18683 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42855
I20260929 20:51:23.898396 18810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42855 every 8 connection(s)
I20260929 20:51:23.899622 18683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:51:23.907670 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 18683
I20260929 20:51:23.907840 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:51:23.908255 388 external_mini_cluster.cc:1428] Setting key d91a088e4b3e56ea3e26879ab51eff2d
Sep 29 20:51:23 dist-test-slave-blv7 krb5kdc[18577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:51:23.919602 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:23.902085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47591 (local address 127.0.97.62:45681)
0929 20:51:23.902657 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:23.902678 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:23.903541 (+ 863us) server_negotiation.cc:407] Connection header received
0929 20:51:23.905327 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:23.905333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:23.905407 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:23.905503 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:23.906980 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:23.907745 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:23.909485 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:23.909960 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:23.910074 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:23.913634 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:23.913641 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:23.913645 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:23.913682 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:23.916522 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:23.917195 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:23.917200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:23.917203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:23.917342 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:23.917754 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:23.917758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:23.917762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:23.918085 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:23.918214 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:23.919170 (+ 956us) server_negotiation.cc:299] Negotiation successful
0929 20:51:23.919360 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":336,"threads_started":1}
I20260929 20:51:23.920950 18814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:23.902630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45681 (local address 127.0.97.1:47591)
0929 20:51:23.903378 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:23.903421 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:23.905051 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:23.905770 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:23.905779 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:23.906273 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:51:23.906810 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:23.906828 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:23.907898 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:23.907902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:23.909353 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:23.909366 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:23.910187 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:23.910195 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:23.910406 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:23.911100 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:51:23.911122 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:51:23.913417 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:51:23.916654 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:51:23.916660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:51:23.916673 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:51:23.917080 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:51:23.917459 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:51:23.917463 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:51:23.917465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:51:23.917647 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:51:23.918271 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:51:23.918279 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:51:23.918707 (+ 428us) client_negotiation.cc:769] Sending connection context
0929 20:51:23.919007 (+ 300us) client_negotiation.cc:240] Negotiation successful
0929 20:51:23.919564 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":252,"threads_started":1}
I20260929 20:51:23.922577 18811 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45681
I20260929 20:51:23.922887 18811 heartbeater.cc:464] Registering TS with master...
I20260929 20:51:23.923626 18811 heartbeater.cc:511] Master 127.0.97.62:45681 requested a full tablet report, sending...
I20260929 20:51:23.925688 18622 ts_manager.cc:194] Registered new tserver with Master: 0b365d7d5d4a40ac8ccbbe1b6610f326 (127.0.97.1:42855)
I20260929 20:51:23.927181 18622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47591
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:51:23.960548 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45681
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:24.271909 18817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:24.272181 18817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:24.272332 18817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:24.279455 18817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:24.279568 18817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:24.279728 18817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:51:24.289299 18817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:24.290661 18817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:24.292110 18817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:24.304747 18822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:24.304754 18823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:24.304672 18825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:24.305976 18817 server_base.cc:1034] running on GCE node
I20260929 20:51:24.306558 18817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:24.307502 18817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:24.308736 18817 hybrid_clock.cc:648] HybridClock initialized: now 1790715084308670 us; error 55 us; skew 500 ppm
I20260929 20:51:24.309101 18817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:24 dist-test-slave-blv7 krb5kdc[18577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:24.312223 18817 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:51:24.313614 18817 webserver.cc:460] Webserver started at http://127.0.97.2:44485/ using document root <none> and password file <none>
I20260929 20:51:24.314282 18817 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:24.314393 18817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:24.314692 18817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:24.318161 18817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "e84f618eeafc4edabe45c2ffd3e56800"
format_stamp: "Formatted at 2026-09-29 20:51:24 on dist-test-slave-blv7"
server_key: "72f0bc3d087e5ad8e01a01ddeec41002"
server_key_iv: "7806f78a3685d41e675a4e79528458fa"
server_key_version: "encryptionkey@0"
I20260929 20:51:24.319243 18817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e84f618eeafc4edabe45c2ffd3e56800"
format_stamp: "Formatted at 2026-09-29 20:51:24 on dist-test-slave-blv7"
server_key: "72f0bc3d087e5ad8e01a01ddeec41002"
server_key_iv: "7806f78a3685d41e675a4e79528458fa"
server_key_version: "encryptionkey@0"
I20260929 20:51:24.326648 18817 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260929 20:51:24.331471 18832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:24.333207 18817 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:51:24.333405 18817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "e84f618eeafc4edabe45c2ffd3e56800"
format_stamp: "Formatted at 2026-09-29 20:51:24 on dist-test-slave-blv7"
server_key: "72f0bc3d087e5ad8e01a01ddeec41002"
server_key_iv: "7806f78a3685d41e675a4e79528458fa"
server_key_version: "encryptionkey@0"
I20260929 20:51:24.333680 18817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:24.361056 18817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:24.365073 18817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:24.365331 18817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:24.366287 18817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:51:24.367599 18817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:51:24.367684 18817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:24.367798 18817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:51:24.367849 18817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:24.397356 18817 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44655
I20260929 20:51:24.397433 18944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44655 every 8 connection(s)
I20260929 20:51:24.398583 18817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:51:24.402029 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 18817
I20260929 20:51:24.402204 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:51:24.402755 388 external_mini_cluster.cc:1428] Setting key 58da9617225470f2ca302bf7c4ee3a28
Sep 29 20:51:24 dist-test-slave-blv7 krb5kdc[18577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:51:24.418490 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:24.401223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46475 (local address 127.0.97.62:45681)
0929 20:51:24.401399 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:24.401403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:24.402544 (+ 1141us) server_negotiation.cc:407] Connection header received
0929 20:51:24.404721 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:24.404727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:24.404806 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:24.404889 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:24.406416 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:24.407149 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:24.408676 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:24.409096 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:24.409205 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:24.412699 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:24.412714 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:24.412719 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:24.412759 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:24.415645 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:24.416298 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:24.416302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:24.416305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:24.416443 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:24.416816 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:24.416821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:24.416825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:24.417121 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:24.417250 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:24.417923 (+ 673us) server_negotiation.cc:299] Negotiation successful
0929 20:51:24.418150 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:51:24.419317 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:24.401523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45681 (local address 127.0.97.2:46475)
0929 20:51:24.402378 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:24.402420 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:24.404435 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:24.405162 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:24.405170 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:24.405677 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:51:24.406239 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:24.406260 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:24.407294 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:24.407298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:24.408518 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:24.408530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:24.409311 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:24.409319 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:24.409535 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:24.410224 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:51:24.410246 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:51:24.412473 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:51:24.415767 (+ 3294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:51:24.415773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:51:24.415785 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:51:24.416188 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:51:24.416531 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:51:24.416534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:51:24.416537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:51:24.416724 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:51:24.417236 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:51:24.417243 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:51:24.417576 (+ 333us) client_negotiation.cc:769] Sending connection context
0929 20:51:24.417807 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 20:51:24.418196 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":312,"threads_started":1}
I20260929 20:51:24.420771 18945 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45681
I20260929 20:51:24.421070 18945 heartbeater.cc:464] Registering TS with master...
I20260929 20:51:24.421774 18945 heartbeater.cc:511] Master 127.0.97.62:45681 requested a full tablet report, sending...
I20260929 20:51:24.423367 18622 ts_manager.cc:194] Registered new tserver with Master: e84f618eeafc4edabe45c2ffd3e56800 (127.0.97.2:44655)
I20260929 20:51:24.424145 18622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46475
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:51:24.459049 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45681
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:24.766851 18950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:24.767150 18950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:24.767304 18950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:24.774690 18950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:24.774799 18950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:24.774955 18950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:51:24.784565 18950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:24.785773 18950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:24.786998 18950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:24.799340 18955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:24.799401 18958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:24.799376 18956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:24.800176 18950 server_base.cc:1034] running on GCE node
I20260929 20:51:24.800729 18950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:24.801680 18950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:24.802888 18950 hybrid_clock.cc:648] HybridClock initialized: now 1790715084802843 us; error 32 us; skew 500 ppm
I20260929 20:51:24.803249 18950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:24 dist-test-slave-blv7 krb5kdc[18577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:24.813990 18950 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:51:24.815876 18950 webserver.cc:460] Webserver started at http://127.0.97.3:44185/ using document root <none> and password file <none>
I20260929 20:51:24.816588 18950 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:24.816717 18950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:24.817025 18950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:24.820308 18950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "c1bf4a585e424e50ad2ea64f53de05d5"
format_stamp: "Formatted at 2026-09-29 20:51:24 on dist-test-slave-blv7"
server_key: "139ec4e6521dcebe789e1f8dd04c4b76"
server_key_iv: "d58c13f26ef190147da6c5bf06ac02f1"
server_key_version: "encryptionkey@0"
I20260929 20:51:24.821177 18950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c1bf4a585e424e50ad2ea64f53de05d5"
format_stamp: "Formatted at 2026-09-29 20:51:24 on dist-test-slave-blv7"
server_key: "139ec4e6521dcebe789e1f8dd04c4b76"
server_key_iv: "d58c13f26ef190147da6c5bf06ac02f1"
server_key_version: "encryptionkey@0"
I20260929 20:51:24.826643 18950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260929 20:51:24.830363 18965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:24.831827 18950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:51:24.831977 18950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "c1bf4a585e424e50ad2ea64f53de05d5"
format_stamp: "Formatted at 2026-09-29 20:51:24 on dist-test-slave-blv7"
server_key: "139ec4e6521dcebe789e1f8dd04c4b76"
server_key_iv: "d58c13f26ef190147da6c5bf06ac02f1"
server_key_version: "encryptionkey@0"
I20260929 20:51:24.832199 18950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:24.856611 18950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:24.861039 18950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:24.861301 18950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:24.862334 18950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:51:24.863686 18950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:51:24.863755 18950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:24.863899 18950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:51:24.863956 18950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:24.893447 18950 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43853
I20260929 20:51:24.893514 19077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43853 every 8 connection(s)
I20260929 20:51:24.894645 18950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:51:24.900909 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 18950
I20260929 20:51:24.901088 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:51:24.901634 388 external_mini_cluster.cc:1428] Setting key 39b4eecc7837e49452b435a7fa66615c
Sep 29 20:51:24 dist-test-slave-blv7 krb5kdc[18577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:51:24.914283 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:24.897158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52343 (local address 127.0.97.62:45681)
0929 20:51:24.897440 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:24.897446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:24.898658 (+ 1212us) server_negotiation.cc:407] Connection header received
0929 20:51:24.900720 (+ 2062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:24.900726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:24.900818 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:24.900947 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:24.902797 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:24.903657 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:24.905147 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:24.905551 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:24.905658 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:24.908899 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:24.908903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:24.908907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:24.908939 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:24.911412 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:24.912083 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:24.912087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:24.912090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:24.912210 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:24.912633 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:24.912637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:24.912639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:24.912911 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:24.913040 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:24.913722 (+ 682us) server_negotiation.cc:299] Negotiation successful
0929 20:51:24.913955 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 20:51:24.915210 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:24.897628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45681 (local address 127.0.97.3:52343)
0929 20:51:24.898456 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:24.898493 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:24.900406 (+ 1913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:24.901216 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:24.901227 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:24.901910 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:51:24.902622 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:24.902642 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:24.903812 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:24.903816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:24.904993 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:24.905005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:24.905762 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:24.905770 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:24.905984 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:24.906560 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:51:24.906580 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:51:24.908691 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:51:24.911541 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:51:24.911550 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:51:24.911567 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:51:24.911967 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:51:24.912350 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:51:24.912353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:51:24.912355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:51:24.912527 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:51:24.913043 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:51:24.913051 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:51:24.913363 (+ 312us) client_negotiation.cc:769] Sending connection context
0929 20:51:24.913596 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 20:51:24.914107 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":316,"threads_started":1}
I20260929 20:51:24.916635 19078 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45681
I20260929 20:51:24.916913 19078 heartbeater.cc:464] Registering TS with master...
I20260929 20:51:24.917593 19078 heartbeater.cc:511] Master 127.0.97.62:45681 requested a full tablet report, sending...
I20260929 20:51:24.918870 18622 ts_manager.cc:194] Registered new tserver with Master: c1bf4a585e424e50ad2ea64f53de05d5 (127.0.97.3:43853)
I20260929 20:51:24.919685 18622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52343
I20260929 20:51:24.929265 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:51:24.930356 18811 heartbeater.cc:503] Master 127.0.97.62:45681 was elected leader, sending a full tablet report...
I20260929 20:51:24.946202 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:24.933285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51094 (local address 127.0.97.62:45681)
0929 20:51:24.933474 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:24.933480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:24.933825 (+ 345us) server_negotiation.cc:407] Connection header received
0929 20:51:24.933975 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:24.933980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:24.934041 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:24.934120 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:24.935423 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:24.936158 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:24.937518 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:24.937935 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:24.938038 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:24.941030 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:24.941037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:24.941042 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:24.941081 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:24.943498 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:24.944260 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:24.944264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:24.944266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:24.944376 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:24.944805 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:24.944808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:24.944810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:24.945113 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:24.945374 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:24.945733 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 20:51:24.945917 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:51:24.950186 18622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:51:24.956827 18622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:51:24.991488 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:24.969218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58474 (local address 127.0.97.3:43853)
0929 20:51:24.981350 (+ 12132us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:24.981360 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:24.982469 (+ 1109us) server_negotiation.cc:407] Connection header received
0929 20:51:24.982573 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:24.982582 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:24.982843 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:24.983061 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:24.984193 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:24.985966 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:24.988869 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:24.990146 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:24.990269 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:24.990506 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:24.990927 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 20:51:24.991138 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11923,"thread_start_us":11751,"threads_started":1}
I20260929 20:51:24.991508 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:24.968849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43853 (local address 127.0.0.1:58474)
0929 20:51:24.982023 (+ 13174us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:24.982072 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:24.982274 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:24.983397 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:24.983406 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:24.983445 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:24.983956 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:24.983971 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:24.986141 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:24.986148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:24.988664 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:24.988681 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:24.990348 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:24.990354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:24.990638 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:24.990716 (+ 78us) client_negotiation.cc:769] Sending connection context
0929 20:51:24.990848 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 20:51:24.991080 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13026,"mutex_wait_us":35,"spinlock_wait_cycles":21376,"thread_start_us":12751,"threads_started":1}
I20260929 20:51:24.994216 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:24.968970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51982 (local address 127.0.97.1:42855)
0929 20:51:24.980953 (+ 11983us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:24.980967 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:24.983842 (+ 2875us) server_negotiation.cc:407] Connection header received
0929 20:51:24.983963 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:24.983973 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:24.984247 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:24.985267 (+ 1020us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:24.985571 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:24.988692 (+ 3121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:24.991197 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:24.992662 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:24.992794 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:24.993040 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:24.993675 (+ 635us) server_negotiation.cc:299] Negotiation successful
0929 20:51:24.993851 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11713,"thread_start_us":11494,"threads_started":1}
I20260929 20:51:24.995085 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:24.968732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42855 (local address 127.0.0.1:51982)
0929 20:51:24.982024 (+ 13292us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:24.982075 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:24.982276 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:24.984478 (+ 2202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:24.984485 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:24.984515 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:24.985022 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:24.985036 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:24.988874 (+ 3838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:24.988880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:24.990972 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:24.990987 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:24.993207 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:24.993214 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:24.993412 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:24.993449 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 20:51:24.994482 (+ 1033us) client_negotiation.cc:240] Negotiation successful
0929 20:51:24.994627 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13013,"spinlock_wait_cycles":2816,"thread_start_us":12671,"threads_started":1}
I20260929 20:51:24.995592 19013 tablet_service.cc:1467] Processing CreateTablet for tablet 92f01c352be84f20893ef101b50b80b2 (DEFAULT_TABLE table=test-table [id=c119bf4f3a3944c1a694af0980af568a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:51:24.997200 19013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92f01c352be84f20893ef101b50b80b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:24.997987 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:24.982393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44655 (local address 127.0.0.1:54914)
0929 20:51:24.988966 (+ 6573us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:24.988993 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:24.989115 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:24.990438 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:24.990443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:24.990473 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:24.990976 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:24.990988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:24.993314 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:24.993319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:24.995327 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:24.995344 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:24.997070 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:24.997078 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:24.997326 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:24.997364 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 20:51:24.997458 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 20:51:24.997593 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6427,"mutex_wait_us":41,"thread_start_us":6803,"threads_started":1}
I20260929 20:51:24.998023 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:24.982645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54914 (local address 127.0.97.2:44655)
0929 20:51:24.989838 (+ 7193us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:24.989847 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:24.989876 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:51:24.989995 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:24.990004 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:24.990223 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:24.990601 (+ 378us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:24.991183 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:24.993061 (+ 1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:24.995475 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:24.996839 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:24.996965 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:24.997146 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:24.997552 (+ 406us) server_negotiation.cc:299] Negotiation successful
0929 20:51:24.997719 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7018,"thread_start_us":6888,"threads_started":1}
I20260929 20:51:24.998744 18746 tablet_service.cc:1467] Processing CreateTablet for tablet 92f01c352be84f20893ef101b50b80b2 (DEFAULT_TABLE table=test-table [id=c119bf4f3a3944c1a694af0980af568a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:51:25.000281 18746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92f01c352be84f20893ef101b50b80b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:25.001680 18880 tablet_service.cc:1467] Processing CreateTablet for tablet 92f01c352be84f20893ef101b50b80b2 (DEFAULT_TABLE table=test-table [id=c119bf4f3a3944c1a694af0980af568a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:51:25.002987 18880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92f01c352be84f20893ef101b50b80b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:25.011334 19096 tablet_bootstrap.cc:492] T 92f01c352be84f20893ef101b50b80b2 P 0b365d7d5d4a40ac8ccbbe1b6610f326: Bootstrap starting.
I20260929 20:51:25.013334 19097 tablet_bootstrap.cc:492] T 92f01c352be84f20893ef101b50b80b2 P c1bf4a585e424e50ad2ea64f53de05d5: Bootstrap starting.
I20260929 20:51:25.015234 19096 tablet_bootstrap.cc:654] T 92f01c352be84f20893ef101b50b80b2 P 0b365d7d5d4a40ac8ccbbe1b6610f326: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:25.016655 19096 log.cc:824] T 92f01c352be84f20893ef101b50b80b2 P 0b365d7d5d4a40ac8ccbbe1b6610f326: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:25.016912 19097 tablet_bootstrap.cc:654] T 92f01c352be84f20893ef101b50b80b2 P c1bf4a585e424e50ad2ea64f53de05d5: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:25.018162 19097 log.cc:824] T 92f01c352be84f20893ef101b50b80b2 P c1bf4a585e424e50ad2ea64f53de05d5: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:25.018975 19098 tablet_bootstrap.cc:492] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800: Bootstrap starting.
I20260929 20:51:25.019950 19096 tablet_bootstrap.cc:492] T 92f01c352be84f20893ef101b50b80b2 P 0b365d7d5d4a40ac8ccbbe1b6610f326: No bootstrap required, opened a new log
I20260929 20:51:25.020262 19096 ts_tablet_manager.cc:1397] T 92f01c352be84f20893ef101b50b80b2 P 0b365d7d5d4a40ac8ccbbe1b6610f326: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260929 20:51:25.020684 19097 tablet_bootstrap.cc:492] T 92f01c352be84f20893ef101b50b80b2 P c1bf4a585e424e50ad2ea64f53de05d5: No bootstrap required, opened a new log
I20260929 20:51:25.020932 19097 ts_tablet_manager.cc:1397] T 92f01c352be84f20893ef101b50b80b2 P c1bf4a585e424e50ad2ea64f53de05d5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 20:51:25.022428 19098 tablet_bootstrap.cc:654] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:25.023649 19098 log.cc:824] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:25.028424 19097 raft_consensus.cc:348] T 92f01c352be84f20893ef101b50b80b2 P c1bf4a585e424e50ad2ea64f53de05d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b365d7d5d4a40ac8ccbbe1b6610f326" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "c1bf4a585e424e50ad2ea64f53de05d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43853 } } peers { permanent_uuid: "e84f618eeafc4edabe45c2ffd3e56800" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44655 } }
I20260929 20:51:25.028841 19097 raft_consensus.cc:374] T 92f01c352be84f20893ef101b50b80b2 P c1bf4a585e424e50ad2ea64f53de05d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:25.028928 19097 raft_consensus.cc:729] T 92f01c352be84f20893ef101b50b80b2 P c1bf4a585e424e50ad2ea64f53de05d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1bf4a585e424e50ad2ea64f53de05d5, State: Initialized, Role: FOLLOWER
I20260929 20:51:25.029513 19097 consensus_queue.cc:260] T 92f01c352be84f20893ef101b50b80b2 P c1bf4a585e424e50ad2ea64f53de05d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b365d7d5d4a40ac8ccbbe1b6610f326" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "c1bf4a585e424e50ad2ea64f53de05d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43853 } } peers { permanent_uuid: "e84f618eeafc4edabe45c2ffd3e56800" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44655 } }
I20260929 20:51:25.029904 19098 tablet_bootstrap.cc:492] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800: No bootstrap required, opened a new log
I20260929 20:51:25.030218 19098 ts_tablet_manager.cc:1397] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260929 20:51:25.031075 19096 raft_consensus.cc:348] T 92f01c352be84f20893ef101b50b80b2 P 0b365d7d5d4a40ac8ccbbe1b6610f326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b365d7d5d4a40ac8ccbbe1b6610f326" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "c1bf4a585e424e50ad2ea64f53de05d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43853 } } peers { permanent_uuid: "e84f618eeafc4edabe45c2ffd3e56800" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44655 } }
I20260929 20:51:25.031553 19096 raft_consensus.cc:374] T 92f01c352be84f20893ef101b50b80b2 P 0b365d7d5d4a40ac8ccbbe1b6610f326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:25.031662 19096 raft_consensus.cc:729] T 92f01c352be84f20893ef101b50b80b2 P 0b365d7d5d4a40ac8ccbbe1b6610f326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b365d7d5d4a40ac8ccbbe1b6610f326, State: Initialized, Role: FOLLOWER
I20260929 20:51:25.032334 19096 consensus_queue.cc:260] T 92f01c352be84f20893ef101b50b80b2 P 0b365d7d5d4a40ac8ccbbe1b6610f326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b365d7d5d4a40ac8ccbbe1b6610f326" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "c1bf4a585e424e50ad2ea64f53de05d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43853 } } peers { permanent_uuid: "e84f618eeafc4edabe45c2ffd3e56800" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44655 } }
I20260929 20:51:25.036566 19078 heartbeater.cc:503] Master 127.0.97.62:45681 was elected leader, sending a full tablet report...
I20260929 20:51:25.037317 19097 ts_tablet_manager.cc:1428] T 92f01c352be84f20893ef101b50b80b2 P c1bf4a585e424e50ad2ea64f53de05d5: Time spent starting tablet: real 0.016s user 0.018s sys 0.000s
I20260929 20:51:25.037331 19096 ts_tablet_manager.cc:1428] T 92f01c352be84f20893ef101b50b80b2 P 0b365d7d5d4a40ac8ccbbe1b6610f326: Time spent starting tablet: real 0.017s user 0.010s sys 0.006s
I20260929 20:51:25.041112 19098 raft_consensus.cc:348] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b365d7d5d4a40ac8ccbbe1b6610f326" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "c1bf4a585e424e50ad2ea64f53de05d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43853 } } peers { permanent_uuid: "e84f618eeafc4edabe45c2ffd3e56800" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44655 } }
I20260929 20:51:25.041577 19098 raft_consensus.cc:374] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:25.041710 19098 raft_consensus.cc:729] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e84f618eeafc4edabe45c2ffd3e56800, State: Initialized, Role: FOLLOWER
I20260929 20:51:25.042290 19098 consensus_queue.cc:260] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b365d7d5d4a40ac8ccbbe1b6610f326" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "c1bf4a585e424e50ad2ea64f53de05d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43853 } } peers { permanent_uuid: "e84f618eeafc4edabe45c2ffd3e56800" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44655 } }
I20260929 20:51:25.043480 18945 heartbeater.cc:503] Master 127.0.97.62:45681 was elected leader, sending a full tablet report...
I20260929 20:51:25.044059 19098 ts_tablet_manager.cc:1428] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
W20260929 20:51:25.146205 19079 tablet.cc:2369] T 92f01c352be84f20893ef101b50b80b2 P c1bf4a585e424e50ad2ea64f53de05d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:51:25.150444 18946 tablet.cc:2369] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:51:25.151676 18812 tablet.cc:2369] T 92f01c352be84f20893ef101b50b80b2 P 0b365d7d5d4a40ac8ccbbe1b6610f326: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:51:25.190907 19104 raft_consensus.cc:482] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:51:25.191224 19104 raft_consensus.cc:504] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b365d7d5d4a40ac8ccbbe1b6610f326" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "c1bf4a585e424e50ad2ea64f53de05d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43853 } } peers { permanent_uuid: "e84f618eeafc4edabe45c2ffd3e56800" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44655 } }
I20260929 20:51:25.192945 19104 leader_election.cc:288] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b365d7d5d4a40ac8ccbbe1b6610f326 (127.0.97.1:42855), c1bf4a585e424e50ad2ea64f53de05d5 (127.0.97.3:43853)
I20260929 20:51:25.199546 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:25.193400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43567 (local address 127.0.97.1:42855)
0929 20:51:25.193897 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:25.193903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:25.194118 (+ 215us) server_negotiation.cc:407] Connection header received
0929 20:51:25.194319 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:25.194323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:25.194391 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:25.194481 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:25.195152 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:25.195895 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:25.197811 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:25.198559 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:25.198675 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:25.198789 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:25.199197 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 20:51:25.199316 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355}
I20260929 20:51:25.199587 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:25.193512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58353 (local address 127.0.97.3:43853)
0929 20:51:25.193715 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:25.193720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:25.194251 (+ 531us) server_negotiation.cc:407] Connection header received
0929 20:51:25.194427 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:25.194431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:25.194506 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:25.194596 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:25.195250 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:25.195956 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:25.197818 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:25.198571 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:25.198678 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:25.198786 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:25.199221 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 20:51:25.199346 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:51:25.199728 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:25.193433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43853 (local address 127.0.97.2:58353)
0929 20:51:25.194143 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:25.194159 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:25.194282 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:25.194659 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:25.194663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:25.194702 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:25.195093 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:25.195107 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:25.196126 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:25.196131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:25.197660 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:25.197672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:25.198811 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:25.198822 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:25.198984 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:25.199086 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 20:51:25.199165 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 20:51:25.199283 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":84,"thread_start_us":468,"threads_started":1}
I20260929 20:51:25.200032 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:25.193203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42855 (local address 127.0.97.2:43567)
0929 20:51:25.193973 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:25.193993 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:25.194158 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:25.194506 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:25.194510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:25.194540 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:25.194977 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:25.194986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:25.196029 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:25.196035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:25.197638 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:25.197658 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:25.198839 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:25.198845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:25.198983 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:25.199060 (+ 77us) client_negotiation.cc:769] Sending connection context
0929 20:51:25.199143 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 20:51:25.199266 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":39,"spinlock_wait_cycles":23808,"thread_start_us":404,"threads_started":1}
I20260929 20:51:25.200760 19033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f01c352be84f20893ef101b50b80b2" candidate_uuid: "e84f618eeafc4edabe45c2ffd3e56800" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1bf4a585e424e50ad2ea64f53de05d5" is_pre_election: true
I20260929 20:51:25.200928 18766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f01c352be84f20893ef101b50b80b2" candidate_uuid: "e84f618eeafc4edabe45c2ffd3e56800" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b365d7d5d4a40ac8ccbbe1b6610f326" is_pre_election: true
I20260929 20:51:25.201244 19033 raft_consensus.cc:2454] T 92f01c352be84f20893ef101b50b80b2 P c1bf4a585e424e50ad2ea64f53de05d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e84f618eeafc4edabe45c2ffd3e56800 in term 0.
I20260929 20:51:25.201396 18766 raft_consensus.cc:2454] T 92f01c352be84f20893ef101b50b80b2 P 0b365d7d5d4a40ac8ccbbe1b6610f326 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e84f618eeafc4edabe45c2ffd3e56800 in term 0.
I20260929 20:51:25.201849 18834 leader_election.cc:302] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1bf4a585e424e50ad2ea64f53de05d5, e84f618eeafc4edabe45c2ffd3e56800; no voters:
I20260929 20:51:25.202203 19104 raft_consensus.cc:2781] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:51:25.202351 19104 raft_consensus.cc:482] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:51:25.202450 19104 raft_consensus.cc:3037] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:25.204523 19104 raft_consensus.cc:504] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b365d7d5d4a40ac8ccbbe1b6610f326" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "c1bf4a585e424e50ad2ea64f53de05d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43853 } } peers { permanent_uuid: "e84f618eeafc4edabe45c2ffd3e56800" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44655 } }
I20260929 20:51:25.205070 19104 leader_election.cc:288] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800 [CANDIDATE]: Term 1 election: Requested vote from peers 0b365d7d5d4a40ac8ccbbe1b6610f326 (127.0.97.1:42855), c1bf4a585e424e50ad2ea64f53de05d5 (127.0.97.3:43853)
I20260929 20:51:25.205490 18766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f01c352be84f20893ef101b50b80b2" candidate_uuid: "e84f618eeafc4edabe45c2ffd3e56800" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b365d7d5d4a40ac8ccbbe1b6610f326"
I20260929 20:51:25.205577 19033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f01c352be84f20893ef101b50b80b2" candidate_uuid: "e84f618eeafc4edabe45c2ffd3e56800" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1bf4a585e424e50ad2ea64f53de05d5"
I20260929 20:51:25.205698 18766 raft_consensus.cc:3037] T 92f01c352be84f20893ef101b50b80b2 P 0b365d7d5d4a40ac8ccbbe1b6610f326 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:25.205760 19033 raft_consensus.cc:3037] T 92f01c352be84f20893ef101b50b80b2 P c1bf4a585e424e50ad2ea64f53de05d5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:25.208714 19033 raft_consensus.cc:2454] T 92f01c352be84f20893ef101b50b80b2 P c1bf4a585e424e50ad2ea64f53de05d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e84f618eeafc4edabe45c2ffd3e56800 in term 1.
I20260929 20:51:25.208714 18766 raft_consensus.cc:2454] T 92f01c352be84f20893ef101b50b80b2 P 0b365d7d5d4a40ac8ccbbe1b6610f326 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e84f618eeafc4edabe45c2ffd3e56800 in term 1.
I20260929 20:51:25.209100 18834 leader_election.cc:302] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1bf4a585e424e50ad2ea64f53de05d5, e84f618eeafc4edabe45c2ffd3e56800; no voters:
I20260929 20:51:25.209352 19104 raft_consensus.cc:2781] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:25.209766 19104 raft_consensus.cc:686] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800 [term 1 LEADER]: Becoming Leader. State: Replica: e84f618eeafc4edabe45c2ffd3e56800, State: Running, Role: LEADER
I20260929 20:51:25.210229 19104 consensus_queue.cc:237] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b365d7d5d4a40ac8ccbbe1b6610f326" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } } peers { permanent_uuid: "c1bf4a585e424e50ad2ea64f53de05d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43853 } } peers { permanent_uuid: "e84f618eeafc4edabe45c2ffd3e56800" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44655 } }
I20260929 20:51:25.214502 18621 catalog_manager.cc:5495] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800 reported cstate change: term changed from 0 to 1, leader changed from <none> to e84f618eeafc4edabe45c2ffd3e56800 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e84f618eeafc4edabe45c2ffd3e56800" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b365d7d5d4a40ac8ccbbe1b6610f326" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1bf4a585e424e50ad2ea64f53de05d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e84f618eeafc4edabe45c2ffd3e56800" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44655 } health_report { overall_health: HEALTHY } } }
I20260929 20:51:25.282619 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:25.277522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54916 (local address 127.0.97.2:44655)
0929 20:51:25.277731 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:25.277736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:25.277779 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:51:25.277992 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:25.277996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:25.278073 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:25.278205 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:51:25.278518 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:25.279305 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:25.280654 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:25.281047 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:25.281135 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:25.281585 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:51:25.282073 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:51:25.282210 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:25.282307 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 20:51:25.282342 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:51:25.290623 19033 raft_consensus.cc:1261] T 92f01c352be84f20893ef101b50b80b2 P c1bf4a585e424e50ad2ea64f53de05d5 [term 1 FOLLOWER]: Refusing update from remote peer e84f618eeafc4edabe45c2ffd3e56800: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:51:25.290693 18766 raft_consensus.cc:1261] T 92f01c352be84f20893ef101b50b80b2 P 0b365d7d5d4a40ac8ccbbe1b6610f326 [term 1 FOLLOWER]: Refusing update from remote peer e84f618eeafc4edabe45c2ffd3e56800: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:51:25.291612 19107 consensus_queue.cc:1035] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b365d7d5d4a40ac8ccbbe1b6610f326" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:51:25.291965 19104 consensus_queue.cc:1035] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1bf4a585e424e50ad2ea64f53de05d5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:51:25.301625 19110 mvcc.cc:205] Tried to move back new op lower bound from 7334768989341184000 to 7334768989024694272. Current Snapshot: MvccSnapshot[applied={T|T < 7334768989341184000}]
I20260929 20:51:25.302273 19111 mvcc.cc:205] Tried to move back new op lower bound from 7334768989341184000 to 7334768989024694272. Current Snapshot: MvccSnapshot[applied={T|T < 7334768989341184000}]
I20260929 20:51:25.306677 19112 mvcc.cc:205] Tried to move back new op lower bound from 7334768989341184000 to 7334768989024694272. Current Snapshot: MvccSnapshot[applied={T|T < 7334768989341184000}]
I20260929 20:51:25.315889 18621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:51:25.316124 18621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:51:25.319682 18621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 041ea2a5cddb4066adad68932cdf1bf8: Sending DeleteTablet for 3 replicas of tablet 92f01c352be84f20893ef101b50b80b2
I20260929 20:51:25.320778 18746 tablet_service.cc:1514] Processing DeleteTablet for tablet 92f01c352be84f20893ef101b50b80b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:51:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51982
I20260929 20:51:25.320834 19013 tablet_service.cc:1514] Processing DeleteTablet for tablet 92f01c352be84f20893ef101b50b80b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:51:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58474
I20260929 20:51:25.321947 18880 tablet_service.cc:1514] Processing DeleteTablet for tablet 92f01c352be84f20893ef101b50b80b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:51:25 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54914
I20260929 20:51:25.330749 19122 tablet_replica.cc:330] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800: stopping tablet replica
I20260929 20:51:25.332099 19122 raft_consensus.cc:2229] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:51:25.332654 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 18683
I20260929 20:51:25.333305 19122 raft_consensus.cc:2258] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:51:25.337653 19120 tablet_replica.cc:330] T 92f01c352be84f20893ef101b50b80b2 P 0b365d7d5d4a40ac8ccbbe1b6610f326: stopping tablet replica
I20260929 20:51:25.338653 19122 ts_tablet_manager.cc:1905] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:51:25.339178 19120 raft_consensus.cc:2229] T 92f01c352be84f20893ef101b50b80b2 P 0b365d7d5d4a40ac8ccbbe1b6610f326 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:51:25.339691 19120 raft_consensus.cc:2258] T 92f01c352be84f20893ef101b50b80b2 P 0b365d7d5d4a40ac8ccbbe1b6610f326 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:51:25.345027 19120 ts_tablet_manager.cc:1905] T 92f01c352be84f20893ef101b50b80b2 P 0b365d7d5d4a40ac8ccbbe1b6610f326: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:51:25.346892 19122 ts_tablet_manager.cc:1918] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:51:25.347131 19122 log.cc:1196] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/92f01c352be84f20893ef101b50b80b2
I20260929 20:51:25.347738 19122 ts_tablet_manager.cc:1939] T 92f01c352be84f20893ef101b50b80b2 P e84f618eeafc4edabe45c2ffd3e56800: Deleting consensus metadata
I20260929 20:51:25.349675 18608 catalog_manager.cc:4841] TS e84f618eeafc4edabe45c2ffd3e56800 (127.0.97.2:44655): tablet 92f01c352be84f20893ef101b50b80b2 (table test-table [id=c119bf4f3a3944c1a694af0980af568a]) successfully deleted
I20260929 20:51:25.350848 19120 ts_tablet_manager.cc:1918] T 92f01c352be84f20893ef101b50b80b2 P 0b365d7d5d4a40ac8ccbbe1b6610f326: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:51:25.350983 19120 log.cc:1196] T 92f01c352be84f20893ef101b50b80b2 P 0b365d7d5d4a40ac8ccbbe1b6610f326: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/92f01c352be84f20893ef101b50b80b2
I20260929 20:51:25.351373 19120 ts_tablet_manager.cc:1939] T 92f01c352be84f20893ef101b50b80b2 P 0b365d7d5d4a40ac8ccbbe1b6610f326: Deleting consensus metadata
I20260929 20:51:25.352737 18608 catalog_manager.cc:4841] TS 0b365d7d5d4a40ac8ccbbe1b6610f326 (127.0.97.1:42855): tablet 92f01c352be84f20893ef101b50b80b2 (table test-table [id=c119bf4f3a3944c1a694af0980af568a]) successfully deleted
I20260929 20:51:25.353631 19121 tablet_replica.cc:330] T 92f01c352be84f20893ef101b50b80b2 P c1bf4a585e424e50ad2ea64f53de05d5: stopping tablet replica
I20260929 20:51:25.354091 19121 raft_consensus.cc:2229] T 92f01c352be84f20893ef101b50b80b2 P c1bf4a585e424e50ad2ea64f53de05d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:51:25.354611 19121 raft_consensus.cc:2258] T 92f01c352be84f20893ef101b50b80b2 P c1bf4a585e424e50ad2ea64f53de05d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:51:25.357494 19121 ts_tablet_manager.cc:1905] T 92f01c352be84f20893ef101b50b80b2 P c1bf4a585e424e50ad2ea64f53de05d5: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 20:51:25 dist-test-slave-blv7 krb5kdc[18577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:51:25.363567 19121 ts_tablet_manager.cc:1918] T 92f01c352be84f20893ef101b50b80b2 P c1bf4a585e424e50ad2ea64f53de05d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:51:25.363722 19121 log.cc:1196] T 92f01c352be84f20893ef101b50b80b2 P c1bf4a585e424e50ad2ea64f53de05d5: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/92f01c352be84f20893ef101b50b80b2
I20260929 20:51:25.364166 19121 ts_tablet_manager.cc:1939] T 92f01c352be84f20893ef101b50b80b2 P c1bf4a585e424e50ad2ea64f53de05d5: Deleting consensus metadata
I20260929 20:51:25.365687 18607 catalog_manager.cc:4841] TS c1bf4a585e424e50ad2ea64f53de05d5 (127.0.97.3:43853): tablet 92f01c352be84f20893ef101b50b80b2 (table test-table [id=c119bf4f3a3944c1a694af0980af568a]) successfully deleted
I20260929 20:51:25.370414 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:25.333873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.0.97.1:42855)
0929 20:51:25.334044 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:25.334050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:25.353008 (+ 18958us) server_negotiation.cc:407] Connection header received
0929 20:51:25.353209 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:25.353214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:25.353295 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:25.353380 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:25.355088 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:25.355801 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:25.357715 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:25.358374 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:25.358495 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:25.363961 (+ 5466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:25.363980 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:25.364000 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:25.364043 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:25.367736 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:25.368375 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:25.368384 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:25.368388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:25.368539 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:25.368867 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:25.368871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:25.368875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:25.369310 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:25.369602 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:25.369815 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 20:51:25.370032 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:51:25.371160 18806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:25.475960 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 18683
I20260929 20:51:25.490617 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 18817
Sep 29 20:51:25 dist-test-slave-blv7 krb5kdc[18577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:51:25.507683 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:25.491383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54930 (local address 127.0.97.2:44655)
0929 20:51:25.491566 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:25.491572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:25.491592 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:51:25.491709 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:25.491715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:25.491793 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:25.491901 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:25.493058 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:25.494277 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:25.495771 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:25.496465 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:25.496587 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:25.500916 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:25.500929 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:25.500948 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:25.500997 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:25.504792 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:25.505556 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:25.505566 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:25.505569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:25.505737 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:25.506093 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:25.506098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:25.506102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:25.506539 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:25.506844 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:25.507076 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 20:51:25.507313 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:51:25.508421 18940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:25.620311 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 18817
I20260929 20:51:25.635774 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 18950
Sep 29 20:51:25 dist-test-slave-blv7 krb5kdc[18577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:51:25.650288 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:25.636481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58484 (local address 127.0.97.3:43853)
0929 20:51:25.636657 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:25.636661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:25.636683 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:51:25.636873 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:25.636879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:25.636963 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:25.637045 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:25.638267 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:25.638972 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:25.640262 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:25.640642 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:25.640745 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:25.644991 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:25.645004 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:25.645020 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:25.645051 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:25.647823 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:25.648408 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:25.648415 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:25.648417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:25.648553 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:25.648849 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:25.648852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:25.648854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:25.649196 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:25.649395 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:25.649713 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 20:51:25.649914 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:51:25.650899 19073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:25.738227 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 18950
I20260929 20:51:25.751364 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 18590
I20260929 20:51:25.752528 18652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:25.859992 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 18590
2026-09-29T20:51:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4203 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T20:51:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:51:25Z Disabled control of system clock
I20260929 20:51:25.928092 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42463
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:51:26.253594 19131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:26.253891 19131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:26.253979 19131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:26.261432 19131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:51:26.261551 19131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:26.261595 19131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:26.261662 19131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:51:26.261713 19131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:51:26.271795 19131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42463
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:26.273036 19131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:26.274838 19131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:26.286357 19136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:26.286384 19137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:26.286433 19139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:26.287370 19131 server_base.cc:1034] running on GCE node
I20260929 20:51:26.288149 19131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:26.289604 19131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:26.290818 19131 hybrid_clock.cc:648] HybridClock initialized: now 1790715086290755 us; error 49 us; skew 500 ppm
I20260929 20:51:26.291226 19131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:51:26.293522 19131 webserver.cc:460] Webserver started at http://127.0.97.62:34781/ using document root <none> and password file <none>
I20260929 20:51:26.294185 19131 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:26.294293 19131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:26.294632 19131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:26.297621 19131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "6acd63ada0814f369df8141206e5231a"
format_stamp: "Formatted at 2026-09-29 20:51:26 on dist-test-slave-blv7"
server_key: "5591db2be308ccc8eaff01b88f4ad13c"
server_key_iv: "16ae5be3c4adb5cec52f7e8c0948f5d5"
server_key_version: "encryptionkey@0"
I20260929 20:51:26.298432 19131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "6acd63ada0814f369df8141206e5231a"
format_stamp: "Formatted at 2026-09-29 20:51:26 on dist-test-slave-blv7"
server_key: "5591db2be308ccc8eaff01b88f4ad13c"
server_key_iv: "16ae5be3c4adb5cec52f7e8c0948f5d5"
server_key_version: "encryptionkey@0"
I20260929 20:51:26.303714 19131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 20:51:26.307296 19145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:26.308583 19131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:51:26.308770 19131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "6acd63ada0814f369df8141206e5231a"
format_stamp: "Formatted at 2026-09-29 20:51:26 on dist-test-slave-blv7"
server_key: "5591db2be308ccc8eaff01b88f4ad13c"
server_key_iv: "16ae5be3c4adb5cec52f7e8c0948f5d5"
server_key_version: "encryptionkey@0"
I20260929 20:51:26.308966 19131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:26.333747 19131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:26.334601 19131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:26.334878 19131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:26.351272 19131 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42463
I20260929 20:51:26.351294 19196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42463 every 8 connection(s)
I20260929 20:51:26.352502 19131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:51:26.356053 19197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:26.360580 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 19131
I20260929 20:51:26.360757 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:51:26.361316 388 external_mini_cluster.cc:1428] Setting key 7fbbf101c922e6e2c0d52b92a560fb16
I20260929 20:51:26.366788 19197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a: Bootstrap starting.
I20260929 20:51:26.370059 19197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:26.371129 19197 log.cc:824] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:26.373944 19197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a: No bootstrap required, opened a new log
I20260929 20:51:26.374444 19200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:26.363130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51392 (local address 127.0.97.62:42463)
0929 20:51:26.364106 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:26.364120 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:26.364161 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:51:26.365838 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:26.365860 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:26.366285 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:26.366588 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:26.367177 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:26.368471 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:26.370250 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:26.370921 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:26.371096 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:26.372266 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:26.372276 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:26.372294 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:26.372817 (+ 523us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:26.372886 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:26.372905 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:26.372991 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:26.373218 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 20:51:26.373505 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":345,"threads_started":1}
I20260929 20:51:26.381609 19197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6acd63ada0814f369df8141206e5231a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42463 } }
I20260929 20:51:26.381922 19197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:26.382015 19197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6acd63ada0814f369df8141206e5231a, State: Initialized, Role: FOLLOWER
I20260929 20:51:26.382560 19197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6acd63ada0814f369df8141206e5231a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42463 } }
I20260929 20:51:26.382761 19197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:51:26.382881 19197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:51:26.383040 19197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:26.385185 19197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6acd63ada0814f369df8141206e5231a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42463 } }
I20260929 20:51:26.385627 19197 leader_election.cc:302] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6acd63ada0814f369df8141206e5231a; no voters:
I20260929 20:51:26.386138 19197 leader_election.cc:288] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:51:26.386274 19202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:26.386931 19202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a [term 1 LEADER]: Becoming Leader. State: Replica: 6acd63ada0814f369df8141206e5231a, State: Running, Role: LEADER
I20260929 20:51:26.387461 19202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6acd63ada0814f369df8141206e5231a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42463 } }
I20260929 20:51:26.388057 19197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:51:26.390177 19203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6acd63ada0814f369df8141206e5231a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6acd63ada0814f369df8141206e5231a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42463 } } }
I20260929 20:51:26.390202 19204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6acd63ada0814f369df8141206e5231a. Latest consensus state: current_term: 1 leader_uuid: "6acd63ada0814f369df8141206e5231a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6acd63ada0814f369df8141206e5231a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42463 } } }
I20260929 20:51:26.390727 19204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:26.390754 19203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:26.391839 19208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:51:26.395797 19208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:51:26.404076 19208 catalog_manager.cc:1347] Generated new cluster ID: 24d82e5aad374a49937ab26957f5a7ba
I20260929 20:51:26.404183 19208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:51:26.434947 19208 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:51:26.436228 19208 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:51:26.451526 19208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a: Generated new TSK 0
I20260929 20:51:26.452591 19208 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:51:26.465541 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42463
--builtin_ntp_servers=127.0.97.20:34867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:51:26.779075 19221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:26.779341 19221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:26.779457 19221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:26.786443 19221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:26.786530 19221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:26.786675 19221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:51:26.795362 19221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:26.796576 19221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:26.798337 19221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:26.810316 19226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:26.810433 19227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:26.810446 19229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:26.811627 19221 server_base.cc:1034] running on GCE node
I20260929 20:51:26.812172 19221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:26.813047 19221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:26.814240 19221 hybrid_clock.cc:648] HybridClock initialized: now 1790715086814182 us; error 52 us; skew 500 ppm
I20260929 20:51:26.814527 19221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:51:26.816627 19221 webserver.cc:460] Webserver started at http://127.0.97.1:39637/ using document root <none> and password file <none>
I20260929 20:51:26.817255 19221 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:26.817342 19221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:26.817634 19221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:26.820438 19221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "4a3ca15aaff546488b0cb71a5d5ec822"
format_stamp: "Formatted at 2026-09-29 20:51:26 on dist-test-slave-blv7"
server_key: "900fe95c2faeb2095e2600e05f22696c"
server_key_iv: "faea66d93c8523aa3e671caaeb0b3a89"
server_key_version: "encryptionkey@0"
I20260929 20:51:26.821199 19221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4a3ca15aaff546488b0cb71a5d5ec822"
format_stamp: "Formatted at 2026-09-29 20:51:26 on dist-test-slave-blv7"
server_key: "900fe95c2faeb2095e2600e05f22696c"
server_key_iv: "faea66d93c8523aa3e671caaeb0b3a89"
server_key_version: "encryptionkey@0"
I20260929 20:51:26.825847 19221 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 20:51:26.828989 19235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:26.830248 19221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 20:51:26.830384 19221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "4a3ca15aaff546488b0cb71a5d5ec822"
format_stamp: "Formatted at 2026-09-29 20:51:26 on dist-test-slave-blv7"
server_key: "900fe95c2faeb2095e2600e05f22696c"
server_key_iv: "faea66d93c8523aa3e671caaeb0b3a89"
server_key_version: "encryptionkey@0"
I20260929 20:51:26.830567 19221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:26.844769 19221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:26.845506 19221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:26.845750 19221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:26.846616 19221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:51:26.847993 19221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:51:26.848083 19221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:26.848196 19221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:51:26.848263 19221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:26.877100 19221 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45195
I20260929 20:51:26.877142 19347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45195 every 8 connection(s)
I20260929 20:51:26.878316 19221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:51:26.880398 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 19221
I20260929 20:51:26.880523 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:51:26.880959 388 external_mini_cluster.cc:1428] Setting key ba25c37605849823740c2aca75084346
I20260929 20:51:26.884371 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42463
--builtin_ntp_servers=127.0.97.20:34867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:51:26.891554 19350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:26.880799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33539 (local address 127.0.97.62:42463)
0929 20:51:26.881295 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:26.881301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:26.882527 (+ 1226us) server_negotiation.cc:407] Connection header received
0929 20:51:26.884664 (+ 2137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:26.884669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:26.884737 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:26.884823 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:26.886088 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:26.886777 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:26.888732 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:26.889164 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:26.889259 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:26.890651 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:26.890662 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:26.890665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:26.890682 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:26.890721 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:26.890733 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:26.890795 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:26.891101 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 20:51:26.891205 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":226,"threads_started":1}
I20260929 20:51:26.892590 19351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:26.881406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42463 (local address 127.0.97.1:33539)
0929 20:51:26.882385 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:26.882420 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:26.884374 (+ 1954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:26.885074 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:26.885085 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:26.885198 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:51:26.885948 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:26.885969 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:26.886919 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:26.886924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:26.888611 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:26.888624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:26.889369 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:26.889375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:26.889649 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:26.890338 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:51:26.890341 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:51:26.890388 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:51:26.890475 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:51:26.890818 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:51:26.890826 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:51:26.890833 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:51:26.891003 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 20:51:26.891399 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":348,"threads_started":1}
I20260929 20:51:26.894134 19348 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42463
I20260929 20:51:26.894418 19348 heartbeater.cc:464] Registering TS with master...
I20260929 20:51:26.895157 19348 heartbeater.cc:511] Master 127.0.97.62:42463 requested a full tablet report, sending...
I20260929 20:51:26.897408 19162 ts_manager.cc:194] Registered new tserver with Master: 4a3ca15aaff546488b0cb71a5d5ec822 (127.0.97.1:45195)
I20260929 20:51:26.899385 19162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:33539
W20260929 20:51:27.204704 19352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:27.204977 19352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:27.205089 19352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:27.212271 19352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:27.212373 19352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:27.212520 19352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:51:27.221598 19352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:27.222790 19352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:27.224536 19352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:27.237001 19357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:27.236968 19358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:27.237075 19360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:27.237694 19352 server_base.cc:1034] running on GCE node
I20260929 20:51:27.238246 19352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:27.239192 19352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:27.240417 19352 hybrid_clock.cc:648] HybridClock initialized: now 1790715087240368 us; error 45 us; skew 500 ppm
I20260929 20:51:27.240731 19352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:51:27.242851 19352 webserver.cc:460] Webserver started at http://127.0.97.2:40391/ using document root <none> and password file <none>
I20260929 20:51:27.243530 19352 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:27.243637 19352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:27.243924 19352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:27.246603 19352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "51ced9ffc3f64afea4d1f38b4c717d5e"
format_stamp: "Formatted at 2026-09-29 20:51:27 on dist-test-slave-blv7"
server_key: "d25c0a2df3531e3839db15cf7b8b834a"
server_key_iv: "5f6635000593de7fd0d1e86deeeb2370"
server_key_version: "encryptionkey@0"
I20260929 20:51:27.247388 19352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "51ced9ffc3f64afea4d1f38b4c717d5e"
format_stamp: "Formatted at 2026-09-29 20:51:27 on dist-test-slave-blv7"
server_key: "d25c0a2df3531e3839db15cf7b8b834a"
server_key_iv: "5f6635000593de7fd0d1e86deeeb2370"
server_key_version: "encryptionkey@0"
I20260929 20:51:27.252274 19352 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:51:27.255635 19366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:27.257054 19352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:51:27.257205 19352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "51ced9ffc3f64afea4d1f38b4c717d5e"
format_stamp: "Formatted at 2026-09-29 20:51:27 on dist-test-slave-blv7"
server_key: "d25c0a2df3531e3839db15cf7b8b834a"
server_key_iv: "5f6635000593de7fd0d1e86deeeb2370"
server_key_version: "encryptionkey@0"
I20260929 20:51:27.257397 19352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:27.270661 19352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:27.271503 19352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:27.271749 19352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:27.272610 19352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:51:27.273960 19352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:51:27.274048 19352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:27.274153 19352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:51:27.274221 19352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:27.303046 19352 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35035
I20260929 20:51:27.303085 19478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35035 every 8 connection(s)
I20260929 20:51:27.304272 19352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:51:27.309742 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 19352
I20260929 20:51:27.309916 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:51:27.310509 388 external_mini_cluster.cc:1428] Setting key f8762007d979341213f13fe551a1a960
I20260929 20:51:27.313309 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42463
--builtin_ntp_servers=127.0.97.20:34867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:51:27.316884 19350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:27.306623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38107 (local address 127.0.97.62:42463)
0929 20:51:27.306854 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:27.306861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:27.308203 (+ 1342us) server_negotiation.cc:407] Connection header received
0929 20:51:27.309999 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:27.310005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:27.310103 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:27.310234 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:27.311333 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:27.312238 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:27.313903 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:27.314575 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:27.314713 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:27.316033 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:27.316040 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:27.316045 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:27.316075 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:27.316133 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:27.316152 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:27.316246 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:27.316427 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 20:51:27.316525 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 20:51:27.317701 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:27.307185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42463 (local address 127.0.97.2:38107)
0929 20:51:27.308038 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:27.308077 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:27.309683 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:27.310407 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:27.310415 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:27.310500 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:51:27.311151 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:27.311176 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:27.312389 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:27.312395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:27.313734 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:27.313745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:27.314770 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:27.314775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:27.314976 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:27.315711 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:51:27.315713 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:51:27.315752 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:51:27.315843 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:51:27.316239 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:51:27.316244 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:51:27.316249 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:51:27.316378 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 20:51:27.316658 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":267,"threads_started":1}
I20260929 20:51:27.319123 19479 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42463
I20260929 20:51:27.319379 19479 heartbeater.cc:464] Registering TS with master...
I20260929 20:51:27.320062 19479 heartbeater.cc:511] Master 127.0.97.62:42463 requested a full tablet report, sending...
I20260929 20:51:27.321373 19162 ts_manager.cc:194] Registered new tserver with Master: 51ced9ffc3f64afea4d1f38b4c717d5e (127.0.97.2:35035)
I20260929 20:51:27.322111 19162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:38107
W20260929 20:51:27.631476 19482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:27.631973 19482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:27.632170 19482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:27.639823 19482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:27.640031 19482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:27.640216 19482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:51:27.649864 19482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:27.651432 19482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:27.653174 19482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:27.665927 19490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:27.666141 19488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:27.666509 19487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:27.667330 19482 server_base.cc:1034] running on GCE node
I20260929 20:51:27.667917 19482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:27.668955 19482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:27.670173 19482 hybrid_clock.cc:648] HybridClock initialized: now 1790715087670112 us; error 54 us; skew 500 ppm
I20260929 20:51:27.670460 19482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:51:27.672570 19482 webserver.cc:460] Webserver started at http://127.0.97.3:42497/ using document root <none> and password file <none>
I20260929 20:51:27.673205 19482 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:27.673285 19482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:27.673538 19482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:27.676319 19482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "63fb4f1f52b04cc4962e1cc4e6c36332"
format_stamp: "Formatted at 2026-09-29 20:51:27 on dist-test-slave-blv7"
server_key: "04169adb1a258c2c7c8b0c3959bd5668"
server_key_iv: "76b9fb15e6b912d279b5798600bfb335"
server_key_version: "encryptionkey@0"
I20260929 20:51:27.677037 19482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "63fb4f1f52b04cc4962e1cc4e6c36332"
format_stamp: "Formatted at 2026-09-29 20:51:27 on dist-test-slave-blv7"
server_key: "04169adb1a258c2c7c8b0c3959bd5668"
server_key_iv: "76b9fb15e6b912d279b5798600bfb335"
server_key_version: "encryptionkey@0"
I20260929 20:51:27.681809 19482 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 20:51:27.685441 19496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:27.686985 19482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:51:27.687203 19482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "63fb4f1f52b04cc4962e1cc4e6c36332"
format_stamp: "Formatted at 2026-09-29 20:51:27 on dist-test-slave-blv7"
server_key: "04169adb1a258c2c7c8b0c3959bd5668"
server_key_iv: "76b9fb15e6b912d279b5798600bfb335"
server_key_version: "encryptionkey@0"
I20260929 20:51:27.687392 19482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:27.711966 19482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:27.712827 19482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:27.713063 19482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:27.714087 19482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:51:27.715543 19482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:51:27.715631 19482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:27.715708 19482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:51:27.715763 19482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:27.745028 19482 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35365
I20260929 20:51:27.745090 19608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35365 every 8 connection(s)
I20260929 20:51:27.746240 19482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:51:27.746588 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 19482
I20260929 20:51:27.746728 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:51:27.747277 388 external_mini_cluster.cc:1428] Setting key 2e3cb0f1300fa60656a1261373977c42
I20260929 20:51:27.758237 19350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:27.748871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56953 (local address 127.0.97.62:42463)
0929 20:51:27.749080 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:27.749086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:27.750669 (+ 1583us) server_negotiation.cc:407] Connection header received
0929 20:51:27.752407 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:27.752411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:27.752476 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:27.752554 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:27.753643 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:27.754351 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:27.755877 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:27.756257 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:27.756347 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:27.757625 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:27.757629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:27.757633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:27.757650 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:27.757680 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:27.757691 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:27.757754 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:27.757932 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 20:51:27.758007 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:51:27.759272 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:27.749518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42463 (local address 127.0.97.3:56953)
0929 20:51:27.750510 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:27.750557 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:27.752187 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:27.752802 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:27.752812 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:27.752909 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:51:27.753479 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:27.753497 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:27.754459 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:27.754463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:27.755744 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:27.755755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:27.756459 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:27.756465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:27.756679 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:27.757360 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:51:27.757363 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:51:27.757402 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:51:27.757480 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:51:27.757769 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:51:27.757777 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:51:27.757784 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:51:27.757918 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 20:51:27.758212 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":308,"threads_started":1}
I20260929 20:51:27.760515 19609 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42463
I20260929 20:51:27.760821 19609 heartbeater.cc:464] Registering TS with master...
I20260929 20:51:27.761508 19609 heartbeater.cc:511] Master 127.0.97.62:42463 requested a full tablet report, sending...
I20260929 20:51:27.762792 19162 ts_manager.cc:194] Registered new tserver with Master: 63fb4f1f52b04cc4962e1cc4e6c36332 (127.0.97.3:35365)
I20260929 20:51:27.763590 19162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:56953
I20260929 20:51:27.774255 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:51:27.783787 19350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:27.777949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56132 (local address 127.0.97.62:42463)
0929 20:51:27.778100 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:27.778105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:27.778508 (+ 403us) server_negotiation.cc:407] Connection header received
0929 20:51:27.778654 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:27.778659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:27.778728 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:27.778808 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:27.779309 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:27.779991 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:27.781310 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:27.781700 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:27.781909 (+ 209us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:27.783064 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:27.783071 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:27.783076 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:27.783098 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:27.783133 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:27.783147 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:27.783282 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:27.783464 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 20:51:27.783517 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 20:51:27.787461 19162 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:51:27.794299 19162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:51:27.817647 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:27.805477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35365 (local address 127.0.0.1:53524)
0929 20:51:27.807663 (+ 2186us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:27.807703 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:27.807878 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:27.808965 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:27.808974 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:27.809013 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:27.809437 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:27.809449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:27.811565 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:27.811571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:27.814114 (+ 2543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:27.814132 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:27.816576 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:27.816583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:27.816844 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:27.816900 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 20:51:27.817076 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 20:51:27.817270 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1959,"thread_start_us":1782,"threads_started":1}
I20260929 20:51:27.819218 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:27.805592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.0.97.3:35365)
0929 20:51:27.806455 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:27.806464 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:27.807817 (+ 1353us) server_negotiation.cc:407] Connection header received
0929 20:51:27.808130 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:27.808140 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:27.808408 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:27.808584 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:27.809657 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:27.811366 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:27.814348 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:27.815991 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:27.816102 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:27.816393 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:27.818341 (+ 1948us) server_negotiation.cc:299] Negotiation successful
0929 20:51:27.818780 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":506,"threads_started":1}
I20260929 20:51:27.824146 19544 tablet_service.cc:1467] Processing CreateTablet for tablet a450d055d7ba4ad388ad47ea51ea482c (DEFAULT_TABLE table=test-table [id=57f9b44b59ba48b296ec003d050e4361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:51:27.825757 19544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a450d055d7ba4ad388ad47ea51ea482c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:27.826889 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:27.808061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35035 (local address 127.0.0.1:54640)
0929 20:51:27.818040 (+ 9979us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:27.818066 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:27.818176 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:27.820903 (+ 2727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:27.820908 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:27.820934 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:27.821144 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:27.821154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:27.823050 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:27.823056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:27.824445 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:27.824457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:27.826025 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:27.826030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:27.826199 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:27.826228 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 20:51:27.826304 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 20:51:27.826448 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9832,"mutex_wait_us":31,"thread_start_us":11702,"threads_started":1}
I20260929 20:51:27.826901 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:27.808456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54640 (local address 127.0.97.2:35035)
0929 20:51:27.820288 (+ 11832us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:27.820297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:27.820324 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:51:27.820418 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:27.820425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:27.820661 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:27.821315 (+ 654us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:27.821517 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:27.822893 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:27.824630 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:27.825884 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:27.826009 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:27.826284 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:27.826398 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 20:51:27.826550 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11673,"thread_start_us":11565,"threads_started":1}
I20260929 20:51:27.827888 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:27.805564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45195 (local address 127.0.0.1:50838)
0929 20:51:27.817023 (+ 11459us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:27.817045 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:27.818773 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:27.819928 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:27.819933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:27.819960 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:27.820260 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:27.820272 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:27.822799 (+ 2527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:27.822804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:27.825052 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:27.825071 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:27.827131 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:27.827138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:27.827305 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:27.827336 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:51:27.827430 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 20:51:27.827549 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11291,"mutex_wait_us":3,"thread_start_us":11638,"threads_started":1}
I20260929 20:51:27.828111 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:27.806148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50838 (local address 127.0.97.1:45195)
0929 20:51:27.807221 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:27.807230 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:27.817170 (+ 9940us) server_negotiation.cc:407] Connection header received
0929 20:51:27.819059 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:27.819070 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:27.819442 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:27.819614 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:27.820414 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:27.822657 (+ 2243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:27.825287 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:27.826651 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:27.826782 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:27.826990 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:27.827502 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 20:51:27.827667 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":916,"thread_start_us":787,"threads_started":1}
I20260929 20:51:27.830870 19414 tablet_service.cc:1467] Processing CreateTablet for tablet a450d055d7ba4ad388ad47ea51ea482c (DEFAULT_TABLE table=test-table [id=57f9b44b59ba48b296ec003d050e4361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:51:27.831384 19283 tablet_service.cc:1467] Processing CreateTablet for tablet a450d055d7ba4ad388ad47ea51ea482c (DEFAULT_TABLE table=test-table [id=57f9b44b59ba48b296ec003d050e4361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:51:27.832298 19414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a450d055d7ba4ad388ad47ea51ea482c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:27.832659 19283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a450d055d7ba4ad388ad47ea51ea482c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:27.838330 19627 tablet_bootstrap.cc:492] T a450d055d7ba4ad388ad47ea51ea482c P 63fb4f1f52b04cc4962e1cc4e6c36332: Bootstrap starting.
I20260929 20:51:27.842145 19627 tablet_bootstrap.cc:654] T a450d055d7ba4ad388ad47ea51ea482c P 63fb4f1f52b04cc4962e1cc4e6c36332: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:27.843097 19627 log.cc:824] T a450d055d7ba4ad388ad47ea51ea482c P 63fb4f1f52b04cc4962e1cc4e6c36332: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:27.845497 19627 tablet_bootstrap.cc:492] T a450d055d7ba4ad388ad47ea51ea482c P 63fb4f1f52b04cc4962e1cc4e6c36332: No bootstrap required, opened a new log
I20260929 20:51:27.845731 19627 ts_tablet_manager.cc:1397] T a450d055d7ba4ad388ad47ea51ea482c P 63fb4f1f52b04cc4962e1cc4e6c36332: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260929 20:51:27.847633 19629 tablet_bootstrap.cc:492] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822: Bootstrap starting.
I20260929 20:51:27.851262 19629 tablet_bootstrap.cc:654] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:27.851732 19630 tablet_bootstrap.cc:492] T a450d055d7ba4ad388ad47ea51ea482c P 51ced9ffc3f64afea4d1f38b4c717d5e: Bootstrap starting.
I20260929 20:51:27.852591 19629 log.cc:824] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:27.854699 19630 tablet_bootstrap.cc:654] T a450d055d7ba4ad388ad47ea51ea482c P 51ced9ffc3f64afea4d1f38b4c717d5e: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:27.855671 19629 tablet_bootstrap.cc:492] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822: No bootstrap required, opened a new log
I20260929 20:51:27.855926 19629 ts_tablet_manager.cc:1397] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 20:51:27.855635 19627 raft_consensus.cc:348] T a450d055d7ba4ad388ad47ea51ea482c P 63fb4f1f52b04cc4962e1cc4e6c36332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63fb4f1f52b04cc4962e1cc4e6c36332" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35365 } } peers { permanent_uuid: "4a3ca15aaff546488b0cb71a5d5ec822" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45195 } } peers { permanent_uuid: "51ced9ffc3f64afea4d1f38b4c717d5e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35035 } }
I20260929 20:51:27.856063 19627 raft_consensus.cc:374] T a450d055d7ba4ad388ad47ea51ea482c P 63fb4f1f52b04cc4962e1cc4e6c36332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:27.856173 19627 raft_consensus.cc:729] T a450d055d7ba4ad388ad47ea51ea482c P 63fb4f1f52b04cc4962e1cc4e6c36332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63fb4f1f52b04cc4962e1cc4e6c36332, State: Initialized, Role: FOLLOWER
I20260929 20:51:27.855840 19630 log.cc:824] T a450d055d7ba4ad388ad47ea51ea482c P 51ced9ffc3f64afea4d1f38b4c717d5e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:27.856837 19627 consensus_queue.cc:260] T a450d055d7ba4ad388ad47ea51ea482c P 63fb4f1f52b04cc4962e1cc4e6c36332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63fb4f1f52b04cc4962e1cc4e6c36332" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35365 } } peers { permanent_uuid: "4a3ca15aaff546488b0cb71a5d5ec822" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45195 } } peers { permanent_uuid: "51ced9ffc3f64afea4d1f38b4c717d5e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35035 } }
I20260929 20:51:27.857921 19609 heartbeater.cc:503] Master 127.0.97.62:42463 was elected leader, sending a full tablet report...
I20260929 20:51:27.858568 19627 ts_tablet_manager.cc:1428] T a450d055d7ba4ad388ad47ea51ea482c P 63fb4f1f52b04cc4962e1cc4e6c36332: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20260929 20:51:27.860666 19630 tablet_bootstrap.cc:492] T a450d055d7ba4ad388ad47ea51ea482c P 51ced9ffc3f64afea4d1f38b4c717d5e: No bootstrap required, opened a new log
I20260929 20:51:27.860944 19630 ts_tablet_manager.cc:1397] T a450d055d7ba4ad388ad47ea51ea482c P 51ced9ffc3f64afea4d1f38b4c717d5e: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260929 20:51:27.866050 19629 raft_consensus.cc:348] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63fb4f1f52b04cc4962e1cc4e6c36332" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35365 } } peers { permanent_uuid: "4a3ca15aaff546488b0cb71a5d5ec822" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45195 } } peers { permanent_uuid: "51ced9ffc3f64afea4d1f38b4c717d5e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35035 } }
I20260929 20:51:27.866504 19629 raft_consensus.cc:374] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:27.866602 19629 raft_consensus.cc:729] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a3ca15aaff546488b0cb71a5d5ec822, State: Initialized, Role: FOLLOWER
I20260929 20:51:27.867225 19629 consensus_queue.cc:260] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63fb4f1f52b04cc4962e1cc4e6c36332" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35365 } } peers { permanent_uuid: "4a3ca15aaff546488b0cb71a5d5ec822" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45195 } } peers { permanent_uuid: "51ced9ffc3f64afea4d1f38b4c717d5e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35035 } }
I20260929 20:51:27.868230 19348 heartbeater.cc:503] Master 127.0.97.62:42463 was elected leader, sending a full tablet report...
I20260929 20:51:27.868938 19629 ts_tablet_manager.cc:1428] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 20:51:27.870352 19630 raft_consensus.cc:348] T a450d055d7ba4ad388ad47ea51ea482c P 51ced9ffc3f64afea4d1f38b4c717d5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63fb4f1f52b04cc4962e1cc4e6c36332" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35365 } } peers { permanent_uuid: "4a3ca15aaff546488b0cb71a5d5ec822" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45195 } } peers { permanent_uuid: "51ced9ffc3f64afea4d1f38b4c717d5e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35035 } }
I20260929 20:51:27.870671 19630 raft_consensus.cc:374] T a450d055d7ba4ad388ad47ea51ea482c P 51ced9ffc3f64afea4d1f38b4c717d5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:27.870764 19630 raft_consensus.cc:729] T a450d055d7ba4ad388ad47ea51ea482c P 51ced9ffc3f64afea4d1f38b4c717d5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ced9ffc3f64afea4d1f38b4c717d5e, State: Initialized, Role: FOLLOWER
I20260929 20:51:27.871362 19630 consensus_queue.cc:260] T a450d055d7ba4ad388ad47ea51ea482c P 51ced9ffc3f64afea4d1f38b4c717d5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63fb4f1f52b04cc4962e1cc4e6c36332" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35365 } } peers { permanent_uuid: "4a3ca15aaff546488b0cb71a5d5ec822" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45195 } } peers { permanent_uuid: "51ced9ffc3f64afea4d1f38b4c717d5e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35035 } }
I20260929 20:51:27.872364 19479 heartbeater.cc:503] Master 127.0.97.62:42463 was elected leader, sending a full tablet report...
I20260929 20:51:27.872887 19630 ts_tablet_manager.cc:1428] T a450d055d7ba4ad388ad47ea51ea482c P 51ced9ffc3f64afea4d1f38b4c717d5e: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260929 20:51:27.880481 19349 tablet.cc:2369] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:51:27.995868 19634 raft_consensus.cc:482] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:51:27.996178 19634 raft_consensus.cc:504] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63fb4f1f52b04cc4962e1cc4e6c36332" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35365 } } peers { permanent_uuid: "4a3ca15aaff546488b0cb71a5d5ec822" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45195 } } peers { permanent_uuid: "51ced9ffc3f64afea4d1f38b4c717d5e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35035 } }
I20260929 20:51:27.997928 19634 leader_election.cc:288] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63fb4f1f52b04cc4962e1cc4e6c36332 (127.0.97.3:35365), 51ced9ffc3f64afea4d1f38b4c717d5e (127.0.97.2:35035)
W20260929 20:51:27.997913 19610 tablet.cc:2369] T a450d055d7ba4ad388ad47ea51ea482c P 63fb4f1f52b04cc4962e1cc4e6c36332: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:51:28.005429 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:27.998271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36041 (local address 127.0.97.3:35365)
0929 20:51:27.998439 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:27.998445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:27.998847 (+ 402us) server_negotiation.cc:407] Connection header received
0929 20:51:27.999133 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:27.999140 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:27.999228 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:27.999352 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:27.999933 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:28.000873 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:28.003312 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:28.004450 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:28.004571 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:28.004699 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:28.005032 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 20:51:28.005166 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 20:51:28.005453 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:27.998203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35365 (local address 127.0.97.1:36041)
0929 20:51:27.998741 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:27.998756 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:27.998899 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:27.999389 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:27.999394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:27.999421 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:27.999798 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:27.999809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:28.001001 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:28.001005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:28.003135 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:28.003150 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:28.004641 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:28.004646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:28.004847 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:28.004883 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 20:51:28.004977 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 20:51:28.005100 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":5,"thread_start_us":284,"threads_started":1}
I20260929 20:51:28.005544 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:27.998921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35035 (local address 127.0.97.1:60473)
0929 20:51:27.999521 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:27.999538 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:27.999655 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:27.999939 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:27.999941 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:27.999962 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:28.000239 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:28.000247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:28.001389 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:28.001394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:28.003363 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:28.003373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:28.004816 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:28.004822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:28.005012 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:28.005040 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 20:51:28.005111 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 20:51:28.005261 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":14,"thread_start_us":298,"threads_started":1}
I20260929 20:51:28.005563 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:27.999098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60473 (local address 127.0.97.2:35035)
0929 20:51:27.999294 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:27.999299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:27.999618 (+ 319us) server_negotiation.cc:407] Connection header received
0929 20:51:27.999760 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:27.999764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:27.999831 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:27.999914 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:28.000371 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:28.001251 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:28.003504 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:28.004632 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:28.004739 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:28.004816 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:28.005176 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 20:51:28.005309 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:51:28.006525 19564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a450d055d7ba4ad388ad47ea51ea482c" candidate_uuid: "4a3ca15aaff546488b0cb71a5d5ec822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63fb4f1f52b04cc4962e1cc4e6c36332" is_pre_election: true
I20260929 20:51:28.006610 19434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a450d055d7ba4ad388ad47ea51ea482c" candidate_uuid: "4a3ca15aaff546488b0cb71a5d5ec822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ced9ffc3f64afea4d1f38b4c717d5e" is_pre_election: true
I20260929 20:51:28.007170 19564 raft_consensus.cc:2454] T a450d055d7ba4ad388ad47ea51ea482c P 63fb4f1f52b04cc4962e1cc4e6c36332 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a3ca15aaff546488b0cb71a5d5ec822 in term 0.
I20260929 20:51:28.007174 19434 raft_consensus.cc:2454] T a450d055d7ba4ad388ad47ea51ea482c P 51ced9ffc3f64afea4d1f38b4c717d5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a3ca15aaff546488b0cb71a5d5ec822 in term 0.
I20260929 20:51:28.007830 19238 leader_election.cc:302] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a3ca15aaff546488b0cb71a5d5ec822, 51ced9ffc3f64afea4d1f38b4c717d5e; no voters:
I20260929 20:51:28.008230 19634 raft_consensus.cc:2781] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:51:28.008363 19634 raft_consensus.cc:482] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:51:28.008466 19634 raft_consensus.cc:3037] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:28.010579 19634 raft_consensus.cc:504] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63fb4f1f52b04cc4962e1cc4e6c36332" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35365 } } peers { permanent_uuid: "4a3ca15aaff546488b0cb71a5d5ec822" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45195 } } peers { permanent_uuid: "51ced9ffc3f64afea4d1f38b4c717d5e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35035 } }
I20260929 20:51:28.011169 19634 leader_election.cc:288] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822 [CANDIDATE]: Term 1 election: Requested vote from peers 63fb4f1f52b04cc4962e1cc4e6c36332 (127.0.97.3:35365), 51ced9ffc3f64afea4d1f38b4c717d5e (127.0.97.2:35035)
I20260929 20:51:28.011656 19564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a450d055d7ba4ad388ad47ea51ea482c" candidate_uuid: "4a3ca15aaff546488b0cb71a5d5ec822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63fb4f1f52b04cc4962e1cc4e6c36332"
I20260929 20:51:28.011744 19434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a450d055d7ba4ad388ad47ea51ea482c" candidate_uuid: "4a3ca15aaff546488b0cb71a5d5ec822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ced9ffc3f64afea4d1f38b4c717d5e"
I20260929 20:51:28.011886 19564 raft_consensus.cc:3037] T a450d055d7ba4ad388ad47ea51ea482c P 63fb4f1f52b04cc4962e1cc4e6c36332 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:28.011945 19434 raft_consensus.cc:3037] T a450d055d7ba4ad388ad47ea51ea482c P 51ced9ffc3f64afea4d1f38b4c717d5e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:28.015187 19564 raft_consensus.cc:2454] T a450d055d7ba4ad388ad47ea51ea482c P 63fb4f1f52b04cc4962e1cc4e6c36332 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a3ca15aaff546488b0cb71a5d5ec822 in term 1.
I20260929 20:51:28.015187 19434 raft_consensus.cc:2454] T a450d055d7ba4ad388ad47ea51ea482c P 51ced9ffc3f64afea4d1f38b4c717d5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a3ca15aaff546488b0cb71a5d5ec822 in term 1.
I20260929 20:51:28.015594 19238 leader_election.cc:302] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a3ca15aaff546488b0cb71a5d5ec822, 63fb4f1f52b04cc4962e1cc4e6c36332; no voters:
I20260929 20:51:28.015878 19634 raft_consensus.cc:2781] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:28.016346 19634 raft_consensus.cc:686] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822 [term 1 LEADER]: Becoming Leader. State: Replica: 4a3ca15aaff546488b0cb71a5d5ec822, State: Running, Role: LEADER
I20260929 20:51:28.016897 19634 consensus_queue.cc:237] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63fb4f1f52b04cc4962e1cc4e6c36332" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35365 } } peers { permanent_uuid: "4a3ca15aaff546488b0cb71a5d5ec822" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45195 } } peers { permanent_uuid: "51ced9ffc3f64afea4d1f38b4c717d5e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35035 } }
I20260929 20:51:28.021440 19162 catalog_manager.cc:5495] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a3ca15aaff546488b0cb71a5d5ec822 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "4a3ca15aaff546488b0cb71a5d5ec822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63fb4f1f52b04cc4962e1cc4e6c36332" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a3ca15aaff546488b0cb71a5d5ec822" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51ced9ffc3f64afea4d1f38b4c717d5e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35035 } health_report { overall_health: UNKNOWN } } }
W20260929 20:51:28.056341 19480 tablet.cc:2369] T a450d055d7ba4ad388ad47ea51ea482c P 51ced9ffc3f64afea4d1f38b4c717d5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:51:28.065128 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:28.059718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50842 (local address 127.0.97.1:45195)
0929 20:51:28.059882 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:28.059887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:28.059994 (+ 107us) server_negotiation.cc:407] Connection header received
0929 20:51:28.060246 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:28.060249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:28.060342 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:28.060504 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:51:28.060918 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:28.061635 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:28.062996 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:28.063517 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:28.063643 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:28.064126 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:51:28.064621 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:51:28.064762 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:28.064843 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 20:51:28.064892 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 20:51:28.073094 19564 raft_consensus.cc:1261] T a450d055d7ba4ad388ad47ea51ea482c P 63fb4f1f52b04cc4962e1cc4e6c36332 [term 1 FOLLOWER]: Refusing update from remote peer 4a3ca15aaff546488b0cb71a5d5ec822: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:51:28.073086 19434 raft_consensus.cc:1261] T a450d055d7ba4ad388ad47ea51ea482c P 51ced9ffc3f64afea4d1f38b4c717d5e [term 1 FOLLOWER]: Refusing update from remote peer 4a3ca15aaff546488b0cb71a5d5ec822: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:51:28.073990 19638 consensus_queue.cc:1035] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ced9ffc3f64afea4d1f38b4c717d5e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:51:28.074325 19634 consensus_queue.cc:1035] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63fb4f1f52b04cc4962e1cc4e6c36332" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:51:28.098805 19161 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:51:28.099261 19161 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:51:28.102905 19161 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6acd63ada0814f369df8141206e5231a: Sending DeleteTablet for 3 replicas of tablet a450d055d7ba4ad388ad47ea51ea482c
I20260929 20:51:28.104008 19414 tablet_service.cc:1514] Processing DeleteTablet for tablet a450d055d7ba4ad388ad47ea51ea482c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:51:28 UTC) from {username='slave'} at 127.0.0.1:54640
I20260929 20:51:28.104058 19283 tablet_service.cc:1514] Processing DeleteTablet for tablet a450d055d7ba4ad388ad47ea51ea482c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:51:28 UTC) from {username='slave'} at 127.0.0.1:50838
I20260929 20:51:28.104794 19651 tablet_replica.cc:330] T a450d055d7ba4ad388ad47ea51ea482c P 51ced9ffc3f64afea4d1f38b4c717d5e: stopping tablet replica
I20260929 20:51:28.104842 19544 tablet_service.cc:1514] Processing DeleteTablet for tablet a450d055d7ba4ad388ad47ea51ea482c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:51:28 UTC) from {username='slave'} at 127.0.0.1:53524
I20260929 20:51:28.105180 19651 raft_consensus.cc:2229] T a450d055d7ba4ad388ad47ea51ea482c P 51ced9ffc3f64afea4d1f38b4c717d5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:51:28.105702 19651 raft_consensus.cc:2258] T a450d055d7ba4ad388ad47ea51ea482c P 51ced9ffc3f64afea4d1f38b4c717d5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:51:28.106007 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 19221
I20260929 20:51:28.107961 19651 ts_tablet_manager.cc:1905] T a450d055d7ba4ad388ad47ea51ea482c P 51ced9ffc3f64afea4d1f38b4c717d5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:51:28.114817 19652 tablet_replica.cc:330] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822: stopping tablet replica
I20260929 20:51:28.115499 19651 ts_tablet_manager.cc:1918] T a450d055d7ba4ad388ad47ea51ea482c P 51ced9ffc3f64afea4d1f38b4c717d5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:51:28.115682 19651 log.cc:1196] T a450d055d7ba4ad388ad47ea51ea482c P 51ced9ffc3f64afea4d1f38b4c717d5e: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/a450d055d7ba4ad388ad47ea51ea482c
I20260929 20:51:28.115835 19652 raft_consensus.cc:2229] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:51:28.116241 19651 ts_tablet_manager.cc:1939] T a450d055d7ba4ad388ad47ea51ea482c P 51ced9ffc3f64afea4d1f38b4c717d5e: Deleting consensus metadata
I20260929 20:51:28.116796 19653 tablet_replica.cc:330] T a450d055d7ba4ad388ad47ea51ea482c P 63fb4f1f52b04cc4962e1cc4e6c36332: stopping tablet replica
I20260929 20:51:28.117513 19652 raft_consensus.cc:2258] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:51:28.118055 19148 catalog_manager.cc:4841] TS 51ced9ffc3f64afea4d1f38b4c717d5e (127.0.97.2:35035): tablet a450d055d7ba4ad388ad47ea51ea482c (table test-table [id=57f9b44b59ba48b296ec003d050e4361]) successfully deleted
I20260929 20:51:28.118496 19653 raft_consensus.cc:2229] T a450d055d7ba4ad388ad47ea51ea482c P 63fb4f1f52b04cc4962e1cc4e6c36332 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:51:28.118886 19653 raft_consensus.cc:2258] T a450d055d7ba4ad388ad47ea51ea482c P 63fb4f1f52b04cc4962e1cc4e6c36332 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:51:28.121047 19653 ts_tablet_manager.cc:1905] T a450d055d7ba4ad388ad47ea51ea482c P 63fb4f1f52b04cc4962e1cc4e6c36332: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:51:28.121054 19652 ts_tablet_manager.cc:1905] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:51:28.127609 19653 ts_tablet_manager.cc:1918] T a450d055d7ba4ad388ad47ea51ea482c P 63fb4f1f52b04cc4962e1cc4e6c36332: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:51:28.127768 19653 log.cc:1196] T a450d055d7ba4ad388ad47ea51ea482c P 63fb4f1f52b04cc4962e1cc4e6c36332: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/a450d055d7ba4ad388ad47ea51ea482c
I20260929 20:51:28.128165 19652 ts_tablet_manager.cc:1918] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:51:28.128350 19653 ts_tablet_manager.cc:1939] T a450d055d7ba4ad388ad47ea51ea482c P 63fb4f1f52b04cc4962e1cc4e6c36332: Deleting consensus metadata
I20260929 20:51:28.128376 19652 log.cc:1196] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/a450d055d7ba4ad388ad47ea51ea482c
I20260929 20:51:28.128916 19652 ts_tablet_manager.cc:1939] T a450d055d7ba4ad388ad47ea51ea482c P 4a3ca15aaff546488b0cb71a5d5ec822: Deleting consensus metadata
I20260929 20:51:28.129992 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:28.109552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.0.97.1:45195)
0929 20:51:28.109746 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:28.109751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:28.120822 (+ 11071us) server_negotiation.cc:407] Connection header received
0929 20:51:28.121281 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:28.121287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:28.121386 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:28.121466 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:28.122136 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:28.123480 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:28.125409 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:28.126080 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:28.126179 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:28.128229 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:28.128240 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:28.128259 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:28.129121 (+ 862us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:28.129182 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:28.129214 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:28.129289 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:28.129524 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 20:51:28.129627 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:51:28.130250 19148 catalog_manager.cc:4841] TS 63fb4f1f52b04cc4962e1cc4e6c36332 (127.0.97.3:35365): tablet a450d055d7ba4ad388ad47ea51ea482c (table test-table [id=57f9b44b59ba48b296ec003d050e4361]) successfully deleted
I20260929 20:51:28.130511 19147 catalog_manager.cc:4841] TS 4a3ca15aaff546488b0cb71a5d5ec822 (127.0.97.1:45195): tablet a450d055d7ba4ad388ad47ea51ea482c (table test-table [id=57f9b44b59ba48b296ec003d050e4361]) successfully deleted
I20260929 20:51:28.131356 19343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:28.225255 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 19221
I20260929 20:51:28.240077 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 19352
I20260929 20:51:28.246641 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:28.240783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54646 (local address 127.0.97.2:35035)
0929 20:51:28.240946 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:28.240950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:28.240971 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:51:28.241135 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:28.241139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:28.241231 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:28.241328 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:28.241781 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:28.242506 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:28.243905 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:28.244292 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:28.244377 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:28.245483 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:28.245490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:28.245505 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:28.246047 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:28.246090 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:28.246108 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:28.246193 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:28.246318 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 20:51:28.246397 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:51:28.247308 19474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:28.336148 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 19352
I20260929 20:51:28.349671 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 19482
I20260929 20:51:28.357026 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:28.350443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53528 (local address 127.0.97.3:35365)
0929 20:51:28.350614 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:28.350619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:28.350685 (+ 66us) server_negotiation.cc:407] Connection header received
0929 20:51:28.350868 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:28.350873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:28.350970 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:28.351080 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:28.351528 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:28.352246 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:28.354125 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:28.354494 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:28.354580 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:28.355770 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:28.355779 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:28.355793 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:28.356328 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:28.356374 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:28.356393 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:28.356474 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:28.356649 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 20:51:28.356737 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:51:28.357726 19604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:28.442793 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 19482
I20260929 20:51:28.457645 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 19131
I20260929 20:51:28.458721 19192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:28.563661 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 19131
2026-09-29T20:51:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2702 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:51:28 dist-test-slave-blv7 krb5kdc[19661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:51:28 dist-test-slave-blv7 krb5kdc[19661](info): set up 2 sockets
Sep 29 20:51:28 dist-test-slave-blv7 krb5kdc[19661](info): commencing operation
krb5kdc: starting...
W20260929 20:51:29.570951 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:51:29 dist-test-slave-blv7 krb5kdc[19661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:51:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:51:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:51:29.791431 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34673
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:30.107419 19674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:30.107704 19674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:30.107831 19674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:30.115039 19674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:51:30.115163 19674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:30.115237 19674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:30.115307 19674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:51:30.115365 19674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:51:30.125313 19674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:30.126590 19674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:30.127871 19674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:30.139114 19680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:30.139115 19682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:30.139185 19679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:30.140082 19674 server_base.cc:1034] running on GCE node
I20260929 20:51:30.140882 19674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:30.142350 19674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:30.143555 19674 hybrid_clock.cc:648] HybridClock initialized: now 1790715090143487 us; error 49 us; skew 500 ppm
I20260929 20:51:30.143877 19674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:30 dist-test-slave-blv7 krb5kdc[19661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:30.150791 19674 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:51:30.152288 19674 webserver.cc:460] Webserver started at http://127.0.97.62:35229/ using document root <none> and password file <none>
I20260929 20:51:30.152938 19674 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:30.153038 19674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:30.153293 19674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:30.156239 19674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "c5b054f17c1f4c188f696f2882fd34c8"
format_stamp: "Formatted at 2026-09-29 20:51:30 on dist-test-slave-blv7"
server_key: "a6304838bf4a89c0e398222f231f89ee"
server_key_iv: "1179970a9a4cc95cdb39029cbcc10bef"
server_key_version: "encryptionkey@0"
I20260929 20:51:30.156920 19674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "c5b054f17c1f4c188f696f2882fd34c8"
format_stamp: "Formatted at 2026-09-29 20:51:30 on dist-test-slave-blv7"
server_key: "a6304838bf4a89c0e398222f231f89ee"
server_key_iv: "1179970a9a4cc95cdb39029cbcc10bef"
server_key_version: "encryptionkey@0"
I20260929 20:51:30.162052 19674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:51:30.165582 19689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:30.166951 19674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:51:30.167140 19674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "c5b054f17c1f4c188f696f2882fd34c8"
format_stamp: "Formatted at 2026-09-29 20:51:30 on dist-test-slave-blv7"
server_key: "a6304838bf4a89c0e398222f231f89ee"
server_key_iv: "1179970a9a4cc95cdb39029cbcc10bef"
server_key_version: "encryptionkey@0"
I20260929 20:51:30.167315 19674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:30.181985 19674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:30.186035 19674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:30.186266 19674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:30.203210 19674 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34673
I20260929 20:51:30.203234 19740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34673 every 8 connection(s)
I20260929 20:51:30.204562 19674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:51:30.208122 19741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:30.211184 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 19674
I20260929 20:51:30.211359 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:51:30.211853 388 external_mini_cluster.cc:1428] Setting key 8c1a62129560a3eac9b208050935a3c4
I20260929 20:51:30.219933 19741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8: Bootstrap starting.
I20260929 20:51:30.223332 19741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:30.224259 19741 log.cc:824] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:51:30 dist-test-slave-blv7 krb5kdc[19661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:51:30.227115 19741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8: No bootstrap required, opened a new log
I20260929 20:51:30.233610 19744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:30.213682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.0.97.62:34673)
0929 20:51:30.214417 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:30.214429 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:30.214469 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:51:30.216143 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:30.216165 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:30.216558 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:30.216891 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:30.218549 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:30.219907 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:30.221262 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:30.222268 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:30.222454 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:30.226399 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:30.226410 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:30.226428 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:30.226469 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:30.229653 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:30.230338 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:30.230344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:30.230351 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:30.230525 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:30.230881 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:30.230885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:30.230887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:30.231463 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:30.231756 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:30.232175 (+ 419us) server_negotiation.cc:299] Negotiation successful
0929 20:51:30.232595 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":359,"threads_started":1}
I20260929 20:51:30.235122 19741 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b054f17c1f4c188f696f2882fd34c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34673 } }
I20260929 20:51:30.235481 19741 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:30.235572 19741 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b054f17c1f4c188f696f2882fd34c8, State: Initialized, Role: FOLLOWER
I20260929 20:51:30.236128 19741 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b054f17c1f4c188f696f2882fd34c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34673 } }
I20260929 20:51:30.236370 19741 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:51:30.236515 19741 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:51:30.236685 19741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:30.238772 19741 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b054f17c1f4c188f696f2882fd34c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34673 } }
I20260929 20:51:30.239295 19741 leader_election.cc:302] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5b054f17c1f4c188f696f2882fd34c8; no voters:
I20260929 20:51:30.239830 19741 leader_election.cc:288] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:51:30.239946 19746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:30.240944 19746 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8 [term 1 LEADER]: Becoming Leader. State: Replica: c5b054f17c1f4c188f696f2882fd34c8, State: Running, Role: LEADER
I20260929 20:51:30.241485 19746 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b054f17c1f4c188f696f2882fd34c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34673 } }
I20260929 20:51:30.241945 19741 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:51:30.243954 19748 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5b054f17c1f4c188f696f2882fd34c8. Latest consensus state: current_term: 1 leader_uuid: "c5b054f17c1f4c188f696f2882fd34c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b054f17c1f4c188f696f2882fd34c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34673 } } }
I20260929 20:51:30.244372 19748 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:30.244297 19747 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5b054f17c1f4c188f696f2882fd34c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b054f17c1f4c188f696f2882fd34c8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34673 } } }
I20260929 20:51:30.244620 19747 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:30.246948 19755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:51:30.251921 19755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:51:30.260703 19755 catalog_manager.cc:1347] Generated new cluster ID: f1093d57624940a7bb0f4a63d31ec544
I20260929 20:51:30.260838 19755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:51:30.306262 19755 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:51:30.308272 19755 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:51:30.323081 19755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8: Generated new TSK 0
I20260929 20:51:30.324201 19755 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 20:51:30.374367 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34673
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:30.690024 19767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:30.690302 19767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:30.690402 19767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:30.698558 19767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:30.698666 19767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:30.698784 19767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:51:30.708653 19767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:30.709872 19767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:30.711105 19767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:30.723486 19775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:30.723587 19772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:30.723559 19773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:30.724305 19767 server_base.cc:1034] running on GCE node
I20260929 20:51:30.724857 19767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:30.725736 19767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:30.726933 19767 hybrid_clock.cc:648] HybridClock initialized: now 1790715090726875 us; error 51 us; skew 500 ppm
I20260929 20:51:30.727247 19767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:30 dist-test-slave-blv7 krb5kdc[19661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:30.730094 19767 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 20:51:30.731451 19767 webserver.cc:460] Webserver started at http://127.0.97.1:45633/ using document root <none> and password file <none>
I20260929 20:51:30.732077 19767 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:30.732167 19767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:30.732406 19767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:30.735314 19767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "89af26a306d34f48a2367348c9f7c713"
format_stamp: "Formatted at 2026-09-29 20:51:30 on dist-test-slave-blv7"
server_key: "aef3682476ed418518803c5c3bf4fc0e"
server_key_iv: "9ba94f996faaca9f8811f2a4993c30a0"
server_key_version: "encryptionkey@0"
I20260929 20:51:30.736028 19767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "89af26a306d34f48a2367348c9f7c713"
format_stamp: "Formatted at 2026-09-29 20:51:30 on dist-test-slave-blv7"
server_key: "aef3682476ed418518803c5c3bf4fc0e"
server_key_iv: "9ba94f996faaca9f8811f2a4993c30a0"
server_key_version: "encryptionkey@0"
I20260929 20:51:30.740839 19767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:51:30.744293 19782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:30.745643 19767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:51:30.745791 19767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "89af26a306d34f48a2367348c9f7c713"
format_stamp: "Formatted at 2026-09-29 20:51:30 on dist-test-slave-blv7"
server_key: "aef3682476ed418518803c5c3bf4fc0e"
server_key_iv: "9ba94f996faaca9f8811f2a4993c30a0"
server_key_version: "encryptionkey@0"
I20260929 20:51:30.745973 19767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:30.758579 19767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:30.762332 19767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:30.762593 19767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:30.763554 19767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:51:30.764885 19767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:51:30.764954 19767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:30.765069 19767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:51:30.765125 19767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:30.794023 19767 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46199
I20260929 20:51:30.794064 19894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46199 every 8 connection(s)
I20260929 20:51:30.795259 19767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:51:30.805568 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 19767
I20260929 20:51:30.805720 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:51:30.806236 388 external_mini_cluster.cc:1428] Setting key 84d9420e5cc76baf32aa167611ded624
Sep 29 20:51:30 dist-test-slave-blv7 krb5kdc[19661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:51:30.814273 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:30.797689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46819 (local address 127.0.97.62:34673)
0929 20:51:30.798109 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:30.798113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:30.798962 (+ 849us) server_negotiation.cc:407] Connection header received
0929 20:51:30.800782 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:30.800787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:30.800867 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:30.800961 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:30.802397 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:30.803175 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:30.804680 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:30.805067 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:30.805174 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:30.808510 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:30.808515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:30.808518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:30.808550 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:30.810980 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:30.811827 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:30.811833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:30.811837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:30.811994 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:30.812438 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:30.812444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:30.812447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:30.812741 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:30.812889 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:30.813594 (+ 705us) server_negotiation.cc:299] Negotiation successful
0929 20:51:30.813837 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":216,"threads_started":1}
I20260929 20:51:30.815322 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:30.798140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.1:46819)
0929 20:51:30.798814 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:30.798851 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:30.800517 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:30.801198 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:30.801207 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:30.801694 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:51:30.802237 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:30.802254 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:30.803317 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:30.803321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:30.804539 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:30.804551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:30.805281 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:30.805289 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:30.805510 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:30.806193 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:51:30.806215 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:51:30.808307 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:51:30.811197 (+ 2890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:51:30.811208 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:51:30.811226 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:51:30.811705 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:51:30.812115 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:51:30.812120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:51:30.812125 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:51:30.812326 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:51:30.812871 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:51:30.812879 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:51:30.813232 (+ 353us) client_negotiation.cc:769] Sending connection context
0929 20:51:30.813491 (+ 259us) client_negotiation.cc:240] Negotiation successful
0929 20:51:30.814037 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":264,"threads_started":1}
I20260929 20:51:30.816941 19895 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34673
I20260929 20:51:30.817258 19895 heartbeater.cc:464] Registering TS with master...
I20260929 20:51:30.818058 19895 heartbeater.cc:511] Master 127.0.97.62:34673 requested a full tablet report, sending...
I20260929 20:51:30.820350 19706 ts_manager.cc:194] Registered new tserver with Master: 89af26a306d34f48a2367348c9f7c713 (127.0.97.1:46199)
I20260929 20:51:30.822299 19706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46819
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 20:51:30.858434 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34673
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:31.167760 19901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:31.168052 19901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:31.168147 19901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:31.175439 19901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:31.175559 19901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:31.175671 19901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:51:31.185418 19901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:31.186636 19901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:31.187886 19901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:31.200021 19906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:31.200062 19909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:31.200081 19907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:31.200721 19901 server_base.cc:1034] running on GCE node
I20260929 20:51:31.201267 19901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:31.202126 19901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:31.203328 19901 hybrid_clock.cc:648] HybridClock initialized: now 1790715091203271 us; error 38 us; skew 500 ppm
I20260929 20:51:31.203661 19901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:31 dist-test-slave-blv7 krb5kdc[19661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:31.207234 19901 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 20:51:31.208828 19901 webserver.cc:460] Webserver started at http://127.0.97.2:42373/ using document root <none> and password file <none>
I20260929 20:51:31.209606 19901 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:31.209741 19901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:31.210052 19901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:31.213685 19901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "8e1195f0d79c4522bae9e85dd07d77c9"
format_stamp: "Formatted at 2026-09-29 20:51:31 on dist-test-slave-blv7"
server_key: "a4d611b284df50e4673afdb0f0c1c9a8"
server_key_iv: "7acea6097ad026ca96771114ffbf2be7"
server_key_version: "encryptionkey@0"
I20260929 20:51:31.214623 19901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8e1195f0d79c4522bae9e85dd07d77c9"
format_stamp: "Formatted at 2026-09-29 20:51:31 on dist-test-slave-blv7"
server_key: "a4d611b284df50e4673afdb0f0c1c9a8"
server_key_iv: "7acea6097ad026ca96771114ffbf2be7"
server_key_version: "encryptionkey@0"
I20260929 20:51:31.220810 19901 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.002s
I20260929 20:51:31.224571 19916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:31.225843 19901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:51:31.225979 19901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "8e1195f0d79c4522bae9e85dd07d77c9"
format_stamp: "Formatted at 2026-09-29 20:51:31 on dist-test-slave-blv7"
server_key: "a4d611b284df50e4673afdb0f0c1c9a8"
server_key_iv: "7acea6097ad026ca96771114ffbf2be7"
server_key_version: "encryptionkey@0"
I20260929 20:51:31.226145 19901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:31.251412 19901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:31.255354 19901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:31.255622 19901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:31.256618 19901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:51:31.257965 19901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:51:31.258039 19901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:31.258157 19901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:51:31.258209 19901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:31.287549 19901 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40271
I20260929 20:51:31.287611 20028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40271 every 8 connection(s)
I20260929 20:51:31.288743 19901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:51:31.290767 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 19901
I20260929 20:51:31.290930 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:51:31.291414 388 external_mini_cluster.cc:1428] Setting key 8efc3b98aef57ace4d10d79adaebe382
Sep 29 20:51:31 dist-test-slave-blv7 krb5kdc[19661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:51:31.308432 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:31.291092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37449 (local address 127.0.97.62:34673)
0929 20:51:31.291300 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:31.291307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:31.292556 (+ 1249us) server_negotiation.cc:407] Connection header received
0929 20:51:31.294354 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:31.294359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:31.294440 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:31.294528 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:31.296036 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.296782 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.298259 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.298643 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.298749 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:31.302709 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:31.302715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:31.302718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:31.302749 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:31.305176 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:31.306139 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:31.306143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:31.306145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:31.306289 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:31.306751 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:31.306754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:31.306756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:31.306994 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:31.307131 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:31.307962 (+ 831us) server_negotiation.cc:299] Negotiation successful
0929 20:51:31.308203 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:51:31.309739 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:31.291658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.2:37449)
0929 20:51:31.292410 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:31.292454 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:31.294102 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:31.294764 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:31.294772 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:31.295324 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:51:31.295878 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:31.295895 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.296907 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.296912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:31.298116 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:31.298127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.298854 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.298861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:31.299117 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:31.299735 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:51:31.299765 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:51:31.302504 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:51:31.305311 (+ 2807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:51:31.305319 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:51:31.305336 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:51:31.306025 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:51:31.306381 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:51:31.306385 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:51:31.306388 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:51:31.306656 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:51:31.307125 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:51:31.307133 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:51:31.307542 (+ 409us) client_negotiation.cc:769] Sending connection context
0929 20:51:31.307823 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 20:51:31.308330 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":300,"threads_started":1}
I20260929 20:51:31.311172 20029 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34673
I20260929 20:51:31.311507 20029 heartbeater.cc:464] Registering TS with master...
I20260929 20:51:31.312400 20029 heartbeater.cc:511] Master 127.0.97.62:34673 requested a full tablet report, sending...
I20260929 20:51:31.313932 19706 ts_manager.cc:194] Registered new tserver with Master: 8e1195f0d79c4522bae9e85dd07d77c9 (127.0.97.2:40271)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 20:51:31.314719 19706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 20:51:31.343675 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34673
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:31.655536 20034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:31.655818 20034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:31.655962 20034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:31.663108 20034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:31.663242 20034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:31.663403 20034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:51:31.673283 20034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:31.674508 20034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:31.675853 20034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:31.687943 20039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:31.688011 20040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:31.688045 20042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:31.688545 20034 server_base.cc:1034] running on GCE node
I20260929 20:51:31.689098 20034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:31.689971 20034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:31.691160 20034 hybrid_clock.cc:648] HybridClock initialized: now 1790715091691094 us; error 51 us; skew 500 ppm
I20260929 20:51:31.691439 20034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:31 dist-test-slave-blv7 krb5kdc[19661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:31.694259 20034 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 20:51:31.695667 20034 webserver.cc:460] Webserver started at http://127.0.97.3:39277/ using document root <none> and password file <none>
I20260929 20:51:31.696297 20034 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:31.696398 20034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:31.696691 20034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:31.700282 20034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "e8f7b23ac84f4c8a85c13ddab7406cf1"
format_stamp: "Formatted at 2026-09-29 20:51:31 on dist-test-slave-blv7"
server_key: "6a209c5e48822c40973308a80284e016"
server_key_iv: "20f184f2b60915084ce34bbcb1cdd873"
server_key_version: "encryptionkey@0"
I20260929 20:51:31.700990 20034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e8f7b23ac84f4c8a85c13ddab7406cf1"
format_stamp: "Formatted at 2026-09-29 20:51:31 on dist-test-slave-blv7"
server_key: "6a209c5e48822c40973308a80284e016"
server_key_iv: "20f184f2b60915084ce34bbcb1cdd873"
server_key_version: "encryptionkey@0"
I20260929 20:51:31.705801 20034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:51:31.709095 20049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:31.710382 20034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:51:31.710525 20034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "e8f7b23ac84f4c8a85c13ddab7406cf1"
format_stamp: "Formatted at 2026-09-29 20:51:31 on dist-test-slave-blv7"
server_key: "6a209c5e48822c40973308a80284e016"
server_key_iv: "20f184f2b60915084ce34bbcb1cdd873"
server_key_version: "encryptionkey@0"
I20260929 20:51:31.710700 20034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:31.729864 20034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:31.733595 20034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:31.733860 20034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:31.734741 20034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:51:31.736011 20034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:51:31.736085 20034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:31.736209 20034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:51:31.736261 20034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:31.765255 20034 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41303
I20260929 20:51:31.765318 20161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41303 every 8 connection(s)
I20260929 20:51:31.766464 20034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:51:31.775336 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 20034
I20260929 20:51:31.775501 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:51:31.775996 388 external_mini_cluster.cc:1428] Setting key 400ab67462a8066abd19228228aeca3c
Sep 29 20:51:31 dist-test-slave-blv7 krb5kdc[19661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:51:31.784687 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:31.768809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52833 (local address 127.0.97.62:34673)
0929 20:51:31.768971 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:31.768975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:31.770056 (+ 1081us) server_negotiation.cc:407] Connection header received
0929 20:51:31.771879 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:31.771884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:31.771951 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:31.772036 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:31.773413 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.774113 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.775959 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.776396 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.776489 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:31.780025 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:31.780030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:31.780033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:31.780071 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:31.782266 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:31.782882 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:31.782886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:31.782888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:31.783000 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:31.783366 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:31.783369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:31.783371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:31.783624 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:31.783734 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:31.784297 (+ 563us) server_negotiation.cc:299] Negotiation successful
0929 20:51:31.784456 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:51:31.785606 20164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:31.769259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34673 (local address 127.0.97.3:52833)
0929 20:51:31.769902 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:31.769939 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:31.771628 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:31.772263 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:31.772271 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:31.772750 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:51:31.773284 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:31.773301 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.774276 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.774280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:31.775800 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:31.775814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.776602 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.776608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:31.776829 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:31.777550 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:51:31.777587 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:51:31.779841 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:51:31.782383 (+ 2542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:51:31.782389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:51:31.782402 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:51:31.782797 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:51:31.783103 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:51:31.783106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:51:31.783108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:51:31.783281 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:51:31.783722 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:51:31.783728 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:51:31.784016 (+ 288us) client_negotiation.cc:769] Sending connection context
0929 20:51:31.784227 (+ 211us) client_negotiation.cc:240] Negotiation successful
0929 20:51:31.784584 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":250,"threads_started":1}
I20260929 20:51:31.786810 20162 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34673
I20260929 20:51:31.787091 20162 heartbeater.cc:464] Registering TS with master...
I20260929 20:51:31.787757 20162 heartbeater.cc:511] Master 127.0.97.62:34673 requested a full tablet report, sending...
I20260929 20:51:31.789048 19706 ts_manager.cc:194] Registered new tserver with Master: e8f7b23ac84f4c8a85c13ddab7406cf1 (127.0.97.3:41303)
I20260929 20:51:31.789949 19706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52833
I20260929 20:51:31.790782 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:51:31.809496 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:31.795172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49856 (local address 127.0.97.62:34673)
0929 20:51:31.795358 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:31.795363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:31.798399 (+ 3036us) server_negotiation.cc:407] Connection header received
0929 20:51:31.798560 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:31.798566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:31.798629 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:31.798706 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:31.800136 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.800843 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.802173 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.802577 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.802680 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:31.804865 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:31.804870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:31.804873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:31.804903 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:31.806979 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:31.807569 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:31.807574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:31.807578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:31.807700 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:31.808046 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:31.808051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:31.808053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:31.808350 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:31.808573 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:31.808899 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 20:51:31.809116 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:51:31.813181 19705 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:51:31.819986 19705 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:51:31.825707 19895 heartbeater.cc:503] Master 127.0.97.62:34673 was elected leader, sending a full tablet report...
I20260929 20:51:31.855212 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:31.831449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53110 (local address 127.0.97.1:46199)
0929 20:51:31.846366 (+ 14917us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:31.846375 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:31.846411 (+ 36us) server_negotiation.cc:407] Connection header received
0929 20:51:31.846537 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:31.846546 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:31.846833 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:31.848295 (+ 1462us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:31.848457 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.849639 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.852705 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.854092 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.854257 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:31.854486 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:31.854707 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 20:51:31.854860 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14737,"thread_start_us":14573,"threads_started":1}
I20260929 20:51:31.855254 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:31.831329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46199 (local address 127.0.0.1:53110)
0929 20:51:31.832259 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:31.832306 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:31.832485 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:31.847641 (+ 15156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:31.847645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:31.847675 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:31.848120 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:31.848130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.849775 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.849780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:31.852533 (+ 2753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:31.852549 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.854301 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.854308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:31.854486 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:31.854538 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 20:51:31.854649 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 20:51:31.854851 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":26,"thread_start_us":16046,"threads_started":1}
I20260929 20:51:31.855706 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:31.831588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40271 (local address 127.0.0.1:42616)
0929 20:51:31.846388 (+ 14800us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:31.846411 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:31.846582 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:31.847498 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:31.847510 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:31.847549 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:31.848117 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:31.848130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.850425 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.850429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:31.852665 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:31.852680 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.854850 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.854856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:31.855050 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:31.855085 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 20:51:31.855165 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 20:51:31.855293 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14680,"mutex_wait_us":2,"thread_start_us":953,"threads_started":1}
I20260929 20:51:31.856475 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:31.832091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42616 (local address 127.0.97.2:40271)
0929 20:51:31.841405 (+ 9314us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:31.841415 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:31.846558 (+ 5143us) server_negotiation.cc:407] Connection header received
0929 20:51:31.846760 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:31.846769 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:31.847064 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:31.848340 (+ 1276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:31.848595 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.850284 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.852802 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.854406 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.854516 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:31.854711 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:31.856019 (+ 1308us) server_negotiation.cc:299] Negotiation successful
0929 20:51:31.856179 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9109,"thread_start_us":8954,"threads_started":1}
I20260929 20:51:31.856501 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:31.833340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41303 (local address 127.0.0.1:33066)
0929 20:51:31.847646 (+ 14306us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:31.847677 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:31.847807 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:31.849900 (+ 2093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:31.849905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:31.849932 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:31.850407 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:31.850419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.851926 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.851931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:31.853796 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:31.853811 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.855752 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.855758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:31.855911 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:31.855941 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:51:31.856026 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 20:51:31.856147 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14172,"mutex_wait_us":3,"spinlock_wait_cycles":1920,"thread_start_us":13346,"threads_started":1}
I20260929 20:51:31.856681 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:31.833692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33066 (local address 127.0.97.3:41303)
0929 20:51:31.849019 (+ 15327us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:31.849055 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:31.849086 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:51:31.849171 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:31.849179 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:31.849479 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:31.849648 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:31.850774 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.851801 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.853952 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.855294 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.855402 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:31.855610 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:31.856082 (+ 472us) server_negotiation.cc:299] Negotiation successful
0929 20:51:31.856245 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15155,"thread_start_us":15418,"threads_started":1}
I20260929 20:51:31.858946 19830 tablet_service.cc:1467] Processing CreateTablet for tablet 9d08c68aed6947ecbd8e20566e222408 (DEFAULT_TABLE table=test-table [id=c0cba50e9cc640ea9b56e0baba9070f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:51:31.859402 19964 tablet_service.cc:1467] Processing CreateTablet for tablet 9d08c68aed6947ecbd8e20566e222408 (DEFAULT_TABLE table=test-table [id=c0cba50e9cc640ea9b56e0baba9070f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:51:31.860316 19830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d08c68aed6947ecbd8e20566e222408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:31.860606 19964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d08c68aed6947ecbd8e20566e222408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:31.863209 20097 tablet_service.cc:1467] Processing CreateTablet for tablet 9d08c68aed6947ecbd8e20566e222408 (DEFAULT_TABLE table=test-table [id=c0cba50e9cc640ea9b56e0baba9070f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:51:31.864527 20097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d08c68aed6947ecbd8e20566e222408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:31.871359 20180 tablet_bootstrap.cc:492] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713: Bootstrap starting.
I20260929 20:51:31.873968 20181 tablet_bootstrap.cc:492] T 9d08c68aed6947ecbd8e20566e222408 P 8e1195f0d79c4522bae9e85dd07d77c9: Bootstrap starting.
I20260929 20:51:31.874461 20180 tablet_bootstrap.cc:654] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:31.875761 20180 log.cc:824] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:31.877249 20181 tablet_bootstrap.cc:654] T 9d08c68aed6947ecbd8e20566e222408 P 8e1195f0d79c4522bae9e85dd07d77c9: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:31.878405 20181 log.cc:824] T 9d08c68aed6947ecbd8e20566e222408 P 8e1195f0d79c4522bae9e85dd07d77c9: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:31.878765 20180 tablet_bootstrap.cc:492] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713: No bootstrap required, opened a new log
I20260929 20:51:31.879139 20180 ts_tablet_manager.cc:1397] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 20:51:31.879307 20183 tablet_bootstrap.cc:492] T 9d08c68aed6947ecbd8e20566e222408 P e8f7b23ac84f4c8a85c13ddab7406cf1: Bootstrap starting.
I20260929 20:51:31.881808 20183 tablet_bootstrap.cc:654] T 9d08c68aed6947ecbd8e20566e222408 P e8f7b23ac84f4c8a85c13ddab7406cf1: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:31.883051 20183 log.cc:824] T 9d08c68aed6947ecbd8e20566e222408 P e8f7b23ac84f4c8a85c13ddab7406cf1: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:31.884285 20181 tablet_bootstrap.cc:492] T 9d08c68aed6947ecbd8e20566e222408 P 8e1195f0d79c4522bae9e85dd07d77c9: No bootstrap required, opened a new log
I20260929 20:51:31.884544 20181 ts_tablet_manager.cc:1397] T 9d08c68aed6947ecbd8e20566e222408 P 8e1195f0d79c4522bae9e85dd07d77c9: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260929 20:51:31.885823 20183 tablet_bootstrap.cc:492] T 9d08c68aed6947ecbd8e20566e222408 P e8f7b23ac84f4c8a85c13ddab7406cf1: No bootstrap required, opened a new log
I20260929 20:51:31.886086 20183 ts_tablet_manager.cc:1397] T 9d08c68aed6947ecbd8e20566e222408 P e8f7b23ac84f4c8a85c13ddab7406cf1: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260929 20:51:31.888263 20180 raft_consensus.cc:348] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af26a306d34f48a2367348c9f7c713" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46199 } } peers { permanent_uuid: "8e1195f0d79c4522bae9e85dd07d77c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40271 } } peers { permanent_uuid: "e8f7b23ac84f4c8a85c13ddab7406cf1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41303 } }
I20260929 20:51:31.888590 20180 raft_consensus.cc:374] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:31.888720 20180 raft_consensus.cc:729] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af26a306d34f48a2367348c9f7c713, State: Initialized, Role: FOLLOWER
I20260929 20:51:31.889250 20180 consensus_queue.cc:260] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af26a306d34f48a2367348c9f7c713" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46199 } } peers { permanent_uuid: "8e1195f0d79c4522bae9e85dd07d77c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40271 } } peers { permanent_uuid: "e8f7b23ac84f4c8a85c13ddab7406cf1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41303 } }
I20260929 20:51:31.890811 20180 ts_tablet_manager.cc:1428] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260929 20:51:31.894565 20181 raft_consensus.cc:348] T 9d08c68aed6947ecbd8e20566e222408 P 8e1195f0d79c4522bae9e85dd07d77c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af26a306d34f48a2367348c9f7c713" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46199 } } peers { permanent_uuid: "8e1195f0d79c4522bae9e85dd07d77c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40271 } } peers { permanent_uuid: "e8f7b23ac84f4c8a85c13ddab7406cf1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41303 } }
I20260929 20:51:31.894999 20181 raft_consensus.cc:374] T 9d08c68aed6947ecbd8e20566e222408 P 8e1195f0d79c4522bae9e85dd07d77c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:31.895133 20181 raft_consensus.cc:729] T 9d08c68aed6947ecbd8e20566e222408 P 8e1195f0d79c4522bae9e85dd07d77c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1195f0d79c4522bae9e85dd07d77c9, State: Initialized, Role: FOLLOWER
I20260929 20:51:31.895890 20181 consensus_queue.cc:260] T 9d08c68aed6947ecbd8e20566e222408 P 8e1195f0d79c4522bae9e85dd07d77c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af26a306d34f48a2367348c9f7c713" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46199 } } peers { permanent_uuid: "8e1195f0d79c4522bae9e85dd07d77c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40271 } } peers { permanent_uuid: "e8f7b23ac84f4c8a85c13ddab7406cf1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41303 } }
I20260929 20:51:31.896008 20183 raft_consensus.cc:348] T 9d08c68aed6947ecbd8e20566e222408 P e8f7b23ac84f4c8a85c13ddab7406cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af26a306d34f48a2367348c9f7c713" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46199 } } peers { permanent_uuid: "8e1195f0d79c4522bae9e85dd07d77c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40271 } } peers { permanent_uuid: "e8f7b23ac84f4c8a85c13ddab7406cf1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41303 } }
I20260929 20:51:31.896425 20183 raft_consensus.cc:374] T 9d08c68aed6947ecbd8e20566e222408 P e8f7b23ac84f4c8a85c13ddab7406cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:31.896520 20183 raft_consensus.cc:729] T 9d08c68aed6947ecbd8e20566e222408 P e8f7b23ac84f4c8a85c13ddab7406cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f7b23ac84f4c8a85c13ddab7406cf1, State: Initialized, Role: FOLLOWER
I20260929 20:51:31.897136 20183 consensus_queue.cc:260] T 9d08c68aed6947ecbd8e20566e222408 P e8f7b23ac84f4c8a85c13ddab7406cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af26a306d34f48a2367348c9f7c713" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46199 } } peers { permanent_uuid: "8e1195f0d79c4522bae9e85dd07d77c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40271 } } peers { permanent_uuid: "e8f7b23ac84f4c8a85c13ddab7406cf1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41303 } }
I20260929 20:51:31.897312 20029 heartbeater.cc:503] Master 127.0.97.62:34673 was elected leader, sending a full tablet report...
I20260929 20:51:31.897625 20181 ts_tablet_manager.cc:1428] T 9d08c68aed6947ecbd8e20566e222408 P 8e1195f0d79c4522bae9e85dd07d77c9: Time spent starting tablet: real 0.013s user 0.006s sys 0.008s
I20260929 20:51:31.898150 20162 heartbeater.cc:503] Master 127.0.97.62:34673 was elected leader, sending a full tablet report...
I20260929 20:51:31.898725 20183 ts_tablet_manager.cc:1428] T 9d08c68aed6947ecbd8e20566e222408 P e8f7b23ac84f4c8a85c13ddab7406cf1: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20260929 20:51:31.931607 20186 raft_consensus.cc:482] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:51:31.931864 20186 raft_consensus.cc:504] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af26a306d34f48a2367348c9f7c713" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46199 } } peers { permanent_uuid: "8e1195f0d79c4522bae9e85dd07d77c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40271 } } peers { permanent_uuid: "e8f7b23ac84f4c8a85c13ddab7406cf1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41303 } }
I20260929 20:51:31.933553 20186 leader_election.cc:288] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e1195f0d79c4522bae9e85dd07d77c9 (127.0.97.2:40271), e8f7b23ac84f4c8a85c13ddab7406cf1 (127.0.97.3:41303)
I20260929 20:51:31.940707 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:31.933926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54551 (local address 127.0.97.2:40271)
0929 20:51:31.934105 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:31.934110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:31.934669 (+ 559us) server_negotiation.cc:407] Connection header received
0929 20:51:31.934977 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:31.934984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:31.935090 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:31.935155 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:31.935769 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.936458 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.938810 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.939925 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.940058 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:31.940153 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:31.940397 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 20:51:31.940521 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:51:31.940913 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:31.933880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40271 (local address 127.0.97.1:54551)
0929 20:51:31.934545 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:31.934563 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:31.934731 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:31.935320 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:31.935323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:31.935353 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:31.935633 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:31.935640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.936580 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.936585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:31.938648 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:31.938663 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.940126 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.940130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:31.940248 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:31.940273 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 20:51:31.940360 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 20:51:31.940491 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":23,"thread_start_us":366,"threads_started":1}
I20260929 20:51:31.941267 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:31.934659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41303 (local address 127.0.97.1:48979)
0929 20:51:31.935238 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:31.935252 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:31.935418 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:31.935703 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:31.935707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:31.935729 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:31.936020 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:31.936028 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.937062 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.937068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:31.938980 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:31.938990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.940464 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.940470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:31.940646 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:31.940680 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:51:31.940782 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 20:51:31.940916 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":22,"spinlock_wait_cycles":22272,"thread_start_us":264,"threads_started":1}
I20260929 20:51:31.941453 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:31.934791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48979 (local address 127.0.97.3:41303)
0929 20:51:31.934976 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:31.934981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:31.935333 (+ 352us) server_negotiation.cc:407] Connection header received
0929 20:51:31.935542 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:31.935546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:31.935601 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:31.935669 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:31.936147 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.936905 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.939127 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.940269 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.940367 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:31.940454 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:31.940834 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 20:51:31.940959 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:51:31.942246 19984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d08c68aed6947ecbd8e20566e222408" candidate_uuid: "89af26a306d34f48a2367348c9f7c713" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1195f0d79c4522bae9e85dd07d77c9" is_pre_election: true
I20260929 20:51:31.942288 20117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d08c68aed6947ecbd8e20566e222408" candidate_uuid: "89af26a306d34f48a2367348c9f7c713" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8f7b23ac84f4c8a85c13ddab7406cf1" is_pre_election: true
I20260929 20:51:31.942833 20117 raft_consensus.cc:2454] T 9d08c68aed6947ecbd8e20566e222408 P e8f7b23ac84f4c8a85c13ddab7406cf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af26a306d34f48a2367348c9f7c713 in term 0.
I20260929 20:51:31.942833 19984 raft_consensus.cc:2454] T 9d08c68aed6947ecbd8e20566e222408 P 8e1195f0d79c4522bae9e85dd07d77c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af26a306d34f48a2367348c9f7c713 in term 0.
I20260929 20:51:31.943506 19786 leader_election.cc:302] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af26a306d34f48a2367348c9f7c713, 8e1195f0d79c4522bae9e85dd07d77c9; no voters:
I20260929 20:51:31.943854 20186 raft_consensus.cc:2781] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:51:31.943991 20186 raft_consensus.cc:482] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:51:31.944087 20186 raft_consensus.cc:3037] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:31.946272 20186 raft_consensus.cc:504] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af26a306d34f48a2367348c9f7c713" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46199 } } peers { permanent_uuid: "8e1195f0d79c4522bae9e85dd07d77c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40271 } } peers { permanent_uuid: "e8f7b23ac84f4c8a85c13ddab7406cf1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41303 } }
I20260929 20:51:31.946880 20186 leader_election.cc:288] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713 [CANDIDATE]: Term 1 election: Requested vote from peers 8e1195f0d79c4522bae9e85dd07d77c9 (127.0.97.2:40271), e8f7b23ac84f4c8a85c13ddab7406cf1 (127.0.97.3:41303)
I20260929 20:51:31.947321 19984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d08c68aed6947ecbd8e20566e222408" candidate_uuid: "89af26a306d34f48a2367348c9f7c713" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e1195f0d79c4522bae9e85dd07d77c9"
I20260929 20:51:31.947374 20117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d08c68aed6947ecbd8e20566e222408" candidate_uuid: "89af26a306d34f48a2367348c9f7c713" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8f7b23ac84f4c8a85c13ddab7406cf1"
I20260929 20:51:31.947499 19984 raft_consensus.cc:3037] T 9d08c68aed6947ecbd8e20566e222408 P 8e1195f0d79c4522bae9e85dd07d77c9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:31.947526 20117 raft_consensus.cc:3037] T 9d08c68aed6947ecbd8e20566e222408 P e8f7b23ac84f4c8a85c13ddab7406cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:31.949608 20117 raft_consensus.cc:2454] T 9d08c68aed6947ecbd8e20566e222408 P e8f7b23ac84f4c8a85c13ddab7406cf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af26a306d34f48a2367348c9f7c713 in term 1.
I20260929 20:51:31.949702 19984 raft_consensus.cc:2454] T 9d08c68aed6947ecbd8e20566e222408 P 8e1195f0d79c4522bae9e85dd07d77c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af26a306d34f48a2367348c9f7c713 in term 1.
I20260929 20:51:31.950023 19786 leader_election.cc:302] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af26a306d34f48a2367348c9f7c713, e8f7b23ac84f4c8a85c13ddab7406cf1; no voters:
I20260929 20:51:31.950269 20186 raft_consensus.cc:2781] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:31.950680 20186 raft_consensus.cc:686] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713 [term 1 LEADER]: Becoming Leader. State: Replica: 89af26a306d34f48a2367348c9f7c713, State: Running, Role: LEADER
I20260929 20:51:31.951201 20186 consensus_queue.cc:237] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af26a306d34f48a2367348c9f7c713" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46199 } } peers { permanent_uuid: "8e1195f0d79c4522bae9e85dd07d77c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40271 } } peers { permanent_uuid: "e8f7b23ac84f4c8a85c13ddab7406cf1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41303 } }
I20260929 20:51:31.955138 19706 catalog_manager.cc:5495] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af26a306d34f48a2367348c9f7c713 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "89af26a306d34f48a2367348c9f7c713" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89af26a306d34f48a2367348c9f7c713" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e1195f0d79c4522bae9e85dd07d77c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8f7b23ac84f4c8a85c13ddab7406cf1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41303 } health_report { overall_health: UNKNOWN } } }
I20260929 20:51:31.981840 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:31.976870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53122 (local address 127.0.97.1:46199)
0929 20:51:31.977070 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:31.977075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:31.977096 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:51:31.977190 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:31.977194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:31.977274 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:31.977395 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:51:31.977762 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.978456 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.979813 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:31.980212 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:31.980322 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:31.980801 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:51:31.981285 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:51:31.981423 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:31.981516 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 20:51:31.981593 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:51:31.989681 19984 raft_consensus.cc:1261] T 9d08c68aed6947ecbd8e20566e222408 P 8e1195f0d79c4522bae9e85dd07d77c9 [term 1 FOLLOWER]: Refusing update from remote peer 89af26a306d34f48a2367348c9f7c713: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:51:31.989691 20117 raft_consensus.cc:1261] T 9d08c68aed6947ecbd8e20566e222408 P e8f7b23ac84f4c8a85c13ddab7406cf1 [term 1 FOLLOWER]: Refusing update from remote peer 89af26a306d34f48a2367348c9f7c713: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:51:31.990598 20191 consensus_queue.cc:1035] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8f7b23ac84f4c8a85c13ddab7406cf1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:51:31.990933 20186 consensus_queue.cc:1035] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e1195f0d79c4522bae9e85dd07d77c9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:51:32.003687 20196 mvcc.cc:205] Tried to move back new op lower bound from 7334769016779911168 to 7334769016635469824. Current Snapshot: MvccSnapshot[applied={T|T < 7334769016779911168}]
I20260929 20:51:32.004925 20195 mvcc.cc:205] Tried to move back new op lower bound from 7334769016779911168 to 7334769016635469824. Current Snapshot: MvccSnapshot[applied={T|T < 7334769016779911168}]
I20260929 20:51:32.012670 19705 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:51:32.012945 19705 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:51:32.018187 19705 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5b054f17c1f4c188f696f2882fd34c8: Sending DeleteTablet for 3 replicas of tablet 9d08c68aed6947ecbd8e20566e222408
I20260929 20:51:32.019434 19830 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d08c68aed6947ecbd8e20566e222408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:51:32 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53110
I20260929 20:51:32.019552 20097 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d08c68aed6947ecbd8e20566e222408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:51:32 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33066
I20260929 20:51:32.019912 20096 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d08c68aed6947ecbd8e20566e222408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:51:32 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33066
I20260929 20:51:32.020231 20204 tablet_replica.cc:330] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713: stopping tablet replica
I20260929 20:51:32.020337 19964 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d08c68aed6947ecbd8e20566e222408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:51:32 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42616
I20260929 20:51:32.020620 20204 raft_consensus.cc:2229] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:51:32.021251 20204 raft_consensus.cc:2258] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:51:32.022089 20205 tablet_replica.cc:330] T 9d08c68aed6947ecbd8e20566e222408 P e8f7b23ac84f4c8a85c13ddab7406cf1: stopping tablet replica
I20260929 20:51:32.022521 20205 raft_consensus.cc:2229] T 9d08c68aed6947ecbd8e20566e222408 P e8f7b23ac84f4c8a85c13ddab7406cf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:51:32.023082 20205 raft_consensus.cc:2258] T 9d08c68aed6947ecbd8e20566e222408 P e8f7b23ac84f4c8a85c13ddab7406cf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:51:32.023557 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 19767
I20260929 20:51:32.025204 20205 ts_tablet_manager.cc:1905] T 9d08c68aed6947ecbd8e20566e222408 P e8f7b23ac84f4c8a85c13ddab7406cf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:51:32.030385 20206 tablet_replica.cc:330] T 9d08c68aed6947ecbd8e20566e222408 P 8e1195f0d79c4522bae9e85dd07d77c9: stopping tablet replica
I20260929 20:51:32.030892 20206 raft_consensus.cc:2229] T 9d08c68aed6947ecbd8e20566e222408 P 8e1195f0d79c4522bae9e85dd07d77c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:51:32.030961 20204 ts_tablet_manager.cc:1905] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:51:32.031432 20206 raft_consensus.cc:2258] T 9d08c68aed6947ecbd8e20566e222408 P 8e1195f0d79c4522bae9e85dd07d77c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:51:32.032459 20205 ts_tablet_manager.cc:1918] T 9d08c68aed6947ecbd8e20566e222408 P e8f7b23ac84f4c8a85c13ddab7406cf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:51:32.032595 20205 log.cc:1196] T 9d08c68aed6947ecbd8e20566e222408 P e8f7b23ac84f4c8a85c13ddab7406cf1: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/9d08c68aed6947ecbd8e20566e222408
I20260929 20:51:32.033084 20205 ts_tablet_manager.cc:1939] T 9d08c68aed6947ecbd8e20566e222408 P e8f7b23ac84f4c8a85c13ddab7406cf1: Deleting consensus metadata
I20260929 20:51:32.033314 20206 ts_tablet_manager.cc:1905] T 9d08c68aed6947ecbd8e20566e222408 P 8e1195f0d79c4522bae9e85dd07d77c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:51:32.034955 19693 catalog_manager.cc:4841] TS e8f7b23ac84f4c8a85c13ddab7406cf1 (127.0.97.3:41303): tablet 9d08c68aed6947ecbd8e20566e222408 (table test-table [id=c0cba50e9cc640ea9b56e0baba9070f3]) successfully deleted
W20260929 20:51:32.035622 19693 catalog_manager.cc:4808] TS e8f7b23ac84f4c8a85c13ddab7406cf1 (127.0.97.3:41303): delete failed for tablet 9d08c68aed6947ecbd8e20566e222408 because the tablet was not found. No further retry: Not found: Tablet not found: 9d08c68aed6947ecbd8e20566e222408
I20260929 20:51:32.039301 20204 ts_tablet_manager.cc:1918] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:51:32.039470 20206 ts_tablet_manager.cc:1918] T 9d08c68aed6947ecbd8e20566e222408 P 8e1195f0d79c4522bae9e85dd07d77c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:51:32.039520 20204 log.cc:1196] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/9d08c68aed6947ecbd8e20566e222408
I20260929 20:51:32.039633 20206 log.cc:1196] T 9d08c68aed6947ecbd8e20566e222408 P 8e1195f0d79c4522bae9e85dd07d77c9: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/9d08c68aed6947ecbd8e20566e222408
I20260929 20:51:32.040125 20204 ts_tablet_manager.cc:1939] T 9d08c68aed6947ecbd8e20566e222408 P 89af26a306d34f48a2367348c9f7c713: Deleting consensus metadata
I20260929 20:51:32.040123 20206 ts_tablet_manager.cc:1939] T 9d08c68aed6947ecbd8e20566e222408 P 8e1195f0d79c4522bae9e85dd07d77c9: Deleting consensus metadata
I20260929 20:51:32.041671 19693 catalog_manager.cc:4841] TS 8e1195f0d79c4522bae9e85dd07d77c9 (127.0.97.2:40271): tablet 9d08c68aed6947ecbd8e20566e222408 (table test-table [id=c0cba50e9cc640ea9b56e0baba9070f3]) successfully deleted
I20260929 20:51:32.041918 19692 catalog_manager.cc:4841] TS 89af26a306d34f48a2367348c9f7c713 (127.0.97.1:46199): tablet 9d08c68aed6947ecbd8e20566e222408 (table test-table [id=c0cba50e9cc640ea9b56e0baba9070f3]) successfully deleted
Sep 29 20:51:32 dist-test-slave-blv7 krb5kdc[19661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 20:51:32.057633 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:32.024832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53128 (local address 127.0.97.1:46199)
0929 20:51:32.024976 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:32.024983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:32.040660 (+ 15677us) server_negotiation.cc:407] Connection header received
0929 20:51:32.041169 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:32.041175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:32.041258 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:32.041341 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:32.043141 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:32.044037 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:32.045956 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:32.046402 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:32.046491 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:32.051186 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:32.051203 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:32.051222 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:32.051266 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:32.054931 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:32.055583 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:32.055593 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:32.055597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:32.055794 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:32.056112 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:32.056118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:32.056121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:32.056539 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:32.056864 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:32.057034 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 20:51:32.057266 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 20:51:32.058348 19890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:32.155931 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 19767
I20260929 20:51:32.170914 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 19901
Sep 29 20:51:32 dist-test-slave-blv7 krb5kdc[19661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 20:51:32.184310 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:32.171610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42620 (local address 127.0.97.2:40271)
0929 20:51:32.171773 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:32.171777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:32.171815 (+ 38us) server_negotiation.cc:407] Connection header received
0929 20:51:32.171961 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:32.171965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:32.172029 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:32.172117 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:32.173211 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:32.173910 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:32.175254 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:32.175631 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:32.175728 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:32.179569 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:32.179585 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:32.179602 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:32.179634 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:32.182098 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:32.182655 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:32.182662 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:32.182664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:32.182770 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:32.183147 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:32.183150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:32.183152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:32.183496 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:32.183722 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:32.183920 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 20:51:32.184071 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:51:32.184935 20024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:32.271091 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 19901
I20260929 20:51:32.284861 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 20034
Sep 29 20:51:32 dist-test-slave-blv7 krb5kdc[19661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 20:51:32.298723 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:32.285641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33078 (local address 127.0.97.3:41303)
0929 20:51:32.285821 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:32.285826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:32.285843 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:51:32.285897 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:32.285901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:32.285984 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:32.286059 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:32.287409 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:32.288103 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:32.289316 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:32.289678 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:32.289760 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:32.293986 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:32.294001 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:32.294016 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:32.294052 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:32.296507 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:32.297069 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:32.297075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:32.297077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:32.297222 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:32.297482 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:32.297485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:32.297487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:32.297805 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:32.298006 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:32.298257 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 20:51:32.298424 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:51:32.299399 20157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:32.385391 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 20034
I20260929 20:51:32.398342 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 19674
I20260929 20:51:32.399387 19736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:32.511650 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 19674
2026-09-29T20:51:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (3949 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T20:51:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:51:32Z Disabled control of system clock
I20260929 20:51:32.582690 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37961
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:51:32.908205 20215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:32.908495 20215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:32.908576 20215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:32.915915 20215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:51:32.916031 20215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:32.916078 20215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:32.916137 20215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:51:32.916179 20215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:51:32.925495 20215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37961
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:32.926755 20215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:32.928555 20215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:32.940016 20220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:32.939965 20223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:32.939934 20221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:32.940941 20215 server_base.cc:1034] running on GCE node
I20260929 20:51:32.941713 20215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:32.943161 20215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:32.944365 20215 hybrid_clock.cc:648] HybridClock initialized: now 1790715092944323 us; error 44 us; skew 500 ppm
I20260929 20:51:32.944693 20215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:51:32.946902 20215 webserver.cc:460] Webserver started at http://127.0.97.62:36435/ using document root <none> and password file <none>
I20260929 20:51:32.947607 20215 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:32.947695 20215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:32.947952 20215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:32.950870 20215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "087a26dbe3484afea52a7c10f95c043b"
format_stamp: "Formatted at 2026-09-29 20:51:32 on dist-test-slave-blv7"
server_key: "7b64abfea66f62dd347d9b9f3bfb8453"
server_key_iv: "e1d27e20b3a35a15287fe831020f3acd"
server_key_version: "encryptionkey@0"
I20260929 20:51:32.951697 20215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "087a26dbe3484afea52a7c10f95c043b"
format_stamp: "Formatted at 2026-09-29 20:51:32 on dist-test-slave-blv7"
server_key: "7b64abfea66f62dd347d9b9f3bfb8453"
server_key_iv: "e1d27e20b3a35a15287fe831020f3acd"
server_key_version: "encryptionkey@0"
I20260929 20:51:32.956634 20215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:51:32.959973 20229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:32.961315 20215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:51:32.961457 20215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "087a26dbe3484afea52a7c10f95c043b"
format_stamp: "Formatted at 2026-09-29 20:51:32 on dist-test-slave-blv7"
server_key: "7b64abfea66f62dd347d9b9f3bfb8453"
server_key_iv: "e1d27e20b3a35a15287fe831020f3acd"
server_key_version: "encryptionkey@0"
I20260929 20:51:32.961623 20215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:32.980188 20215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:32.981024 20215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:32.981253 20215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:32.998337 20215 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37961
I20260929 20:51:32.998360 20280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37961 every 8 connection(s)
I20260929 20:51:32.999656 20215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:51:33.003816 20281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:33.004230 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 20215
I20260929 20:51:33.004362 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:51:33.004848 388 external_mini_cluster.cc:1428] Setting key 514e81d48c4548f71e57b1b511d1ae79
I20260929 20:51:33.017294 20281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b: Bootstrap starting.
I20260929 20:51:33.017710 20284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:33.006432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34118 (local address 127.0.97.62:37961)
0929 20:51:33.007371 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:33.007385 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:33.007428 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:51:33.009339 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:33.009363 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:33.009838 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:33.010247 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:33.010825 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:33.012154 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:33.013409 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:33.014080 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:33.014264 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:33.015416 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:33.015423 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:33.015437 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:33.015956 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:33.016042 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:33.016065 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:33.016159 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:33.016374 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 20:51:33.016672 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":295,"threads_started":1}
I20260929 20:51:33.020570 20281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:33.021554 20281 log.cc:824] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:33.024183 20281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b: No bootstrap required, opened a new log
I20260929 20:51:33.031796 20281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087a26dbe3484afea52a7c10f95c043b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37961 } }
I20260929 20:51:33.032155 20281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:33.032249 20281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 087a26dbe3484afea52a7c10f95c043b, State: Initialized, Role: FOLLOWER
I20260929 20:51:33.032763 20281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087a26dbe3484afea52a7c10f95c043b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37961 } }
I20260929 20:51:33.032974 20281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:51:33.033100 20281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:51:33.033249 20281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:33.035410 20281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087a26dbe3484afea52a7c10f95c043b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37961 } }
I20260929 20:51:33.035890 20281 leader_election.cc:302] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 087a26dbe3484afea52a7c10f95c043b; no voters:
I20260929 20:51:33.036420 20281 leader_election.cc:288] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:51:33.036540 20286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:33.037251 20286 raft_consensus.cc:686] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b [term 1 LEADER]: Becoming Leader. State: Replica: 087a26dbe3484afea52a7c10f95c043b, State: Running, Role: LEADER
I20260929 20:51:33.037724 20286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087a26dbe3484afea52a7c10f95c043b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37961 } }
I20260929 20:51:33.038499 20281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:51:33.040314 20288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 087a26dbe3484afea52a7c10f95c043b. Latest consensus state: current_term: 1 leader_uuid: "087a26dbe3484afea52a7c10f95c043b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087a26dbe3484afea52a7c10f95c043b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37961 } } }
I20260929 20:51:33.040542 20287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "087a26dbe3484afea52a7c10f95c043b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087a26dbe3484afea52a7c10f95c043b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37961 } } }
I20260929 20:51:33.040904 20288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:33.040901 20287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:33.042307 20293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:51:33.047276 20293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:51:33.055219 20293 catalog_manager.cc:1347] Generated new cluster ID: e0ae6f298413468da1b74aad4e62df3a
I20260929 20:51:33.055333 20293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:51:33.071580 20293 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:51:33.073396 20293 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:51:33.083362 20293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b: Generated new TSK 0
I20260929 20:51:33.084345 20293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:51:33.089051 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37961
--builtin_ntp_servers=127.0.97.20:45955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:51:33.408417 20305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:33.408712 20305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:33.408835 20305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:33.415911 20305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:33.416011 20305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:33.416124 20305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 20:51:33.425081 20305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:33.426294 20305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:33.428010 20305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:33.440445 20310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:33.440632 20313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:33.440924 20311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:33.441262 20305 server_base.cc:1034] running on GCE node
I20260929 20:51:33.441875 20305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:33.442862 20305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:33.444103 20305 hybrid_clock.cc:648] HybridClock initialized: now 1790715093444020 us; error 62 us; skew 500 ppm
I20260929 20:51:33.444403 20305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:51:33.446630 20305 webserver.cc:460] Webserver started at http://127.0.97.1:36105/ using document root <none> and password file <none>
I20260929 20:51:33.447364 20305 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:33.447453 20305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:33.447727 20305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:33.450534 20305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-0/data/instance:
uuid: "f8d4c00331ef4193ae8bbe8d224ff9d5"
format_stamp: "Formatted at 2026-09-29 20:51:33 on dist-test-slave-blv7"
server_key: "9527c67713435ed3b5a15c41d7107e19"
server_key_iv: "92b9e37edb6704ad8148be116cd77b24"
server_key_version: "encryptionkey@0"
I20260929 20:51:33.451324 20305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f8d4c00331ef4193ae8bbe8d224ff9d5"
format_stamp: "Formatted at 2026-09-29 20:51:33 on dist-test-slave-blv7"
server_key: "9527c67713435ed3b5a15c41d7107e19"
server_key_iv: "92b9e37edb6704ad8148be116cd77b24"
server_key_version: "encryptionkey@0"
I20260929 20:51:33.456414 20305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 20:51:33.459853 20319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:33.461212 20305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:51:33.461352 20305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-0/wal
uuid: "f8d4c00331ef4193ae8bbe8d224ff9d5"
format_stamp: "Formatted at 2026-09-29 20:51:33 on dist-test-slave-blv7"
server_key: "9527c67713435ed3b5a15c41d7107e19"
server_key_iv: "92b9e37edb6704ad8148be116cd77b24"
server_key_version: "encryptionkey@0"
I20260929 20:51:33.461531 20305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:33.474599 20305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:33.475451 20305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:33.475689 20305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:33.476558 20305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:51:33.477988 20305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:51:33.478070 20305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:33.478148 20305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:51:33.478202 20305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:33.507557 20305 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36407
I20260929 20:51:33.507604 20431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36407 every 8 connection(s)
I20260929 20:51:33.508826 20305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-0/data/info.pb
I20260929 20:51:33.510825 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 20305
I20260929 20:51:33.510969 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-0/wal/instance
I20260929 20:51:33.511471 388 external_mini_cluster.cc:1428] Setting key bf0dec5d396974f99f8b766bfd3a5433
I20260929 20:51:33.514012 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37961
--builtin_ntp_servers=127.0.97.20:45955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:51:33.522322 20284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:33.511420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34507 (local address 127.0.97.62:37961)
0929 20:51:33.511632 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:33.511639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:33.513192 (+ 1553us) server_negotiation.cc:407] Connection header received
0929 20:51:33.514968 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:33.514974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:33.515079 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:33.515184 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:33.516385 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:33.517259 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:33.519384 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:33.519883 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:33.519980 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:33.521535 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:33.521542 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:33.521546 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:33.521569 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:33.521608 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:33.521623 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:33.521696 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:33.521929 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 20:51:33.522008 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:51:33.523370 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:33.511944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37961 (local address 127.0.97.1:34507)
0929 20:51:33.513026 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:33.513068 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:33.514715 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:33.515407 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:33.515415 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:33.515511 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:51:33.516217 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:33.516241 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:33.517427 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:33.517432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:33.519256 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:33.519270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:33.520107 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:33.520114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:33.520380 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:33.521158 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:51:33.521162 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:51:33.521226 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:51:33.521325 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:51:33.521716 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:51:33.521726 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:51:33.521733 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:51:33.521916 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 20:51:33.522291 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":335,"threads_started":1}
I20260929 20:51:33.524979 20432 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37961
I20260929 20:51:33.525281 20432 heartbeater.cc:464] Registering TS with master...
I20260929 20:51:33.525987 20432 heartbeater.cc:511] Master 127.0.97.62:37961 requested a full tablet report, sending...
I20260929 20:51:33.527897 20246 ts_manager.cc:194] Registered new tserver with Master: f8d4c00331ef4193ae8bbe8d224ff9d5 (127.0.97.1:36407)
I20260929 20:51:33.529294 20246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:34507
W20260929 20:51:33.821688 20435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:33.821967 20435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:33.822086 20435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:33.829348 20435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:33.829447 20435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:33.829597 20435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 20:51:33.838635 20435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:33.839936 20435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:33.841619 20435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:33.853792 20440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:33.853807 20441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:33.853968 20443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:33.854533 20435 server_base.cc:1034] running on GCE node
I20260929 20:51:33.855101 20435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:33.855988 20435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:33.857203 20435 hybrid_clock.cc:648] HybridClock initialized: now 1790715093857159 us; error 37 us; skew 500 ppm
I20260929 20:51:33.857488 20435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:51:33.859594 20435 webserver.cc:460] Webserver started at http://127.0.97.2:44517/ using document root <none> and password file <none>
I20260929 20:51:33.860237 20435 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:33.860327 20435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:33.860625 20435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:33.863373 20435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-1/data/instance:
uuid: "3f8101b8a1894e39b76746e9a8e70da9"
format_stamp: "Formatted at 2026-09-29 20:51:33 on dist-test-slave-blv7"
server_key: "a914bfe6e0ede32c7a19dc6acbb5b2a8"
server_key_iv: "5924abe92933fc0307b6d668b51d7183"
server_key_version: "encryptionkey@0"
I20260929 20:51:33.864096 20435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3f8101b8a1894e39b76746e9a8e70da9"
format_stamp: "Formatted at 2026-09-29 20:51:33 on dist-test-slave-blv7"
server_key: "a914bfe6e0ede32c7a19dc6acbb5b2a8"
server_key_iv: "5924abe92933fc0307b6d668b51d7183"
server_key_version: "encryptionkey@0"
I20260929 20:51:33.869094 20435 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:51:33.872342 20449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:33.873600 20435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 20:51:33.873740 20435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-1/wal
uuid: "3f8101b8a1894e39b76746e9a8e70da9"
format_stamp: "Formatted at 2026-09-29 20:51:33 on dist-test-slave-blv7"
server_key: "a914bfe6e0ede32c7a19dc6acbb5b2a8"
server_key_iv: "5924abe92933fc0307b6d668b51d7183"
server_key_version: "encryptionkey@0"
I20260929 20:51:33.873921 20435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:33.886219 20435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:33.886920 20435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:33.887181 20435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:33.888033 20435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:51:33.889344 20435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:51:33.889427 20435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:33.889528 20435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:51:33.889590 20435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:33.917910 20435 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37711
I20260929 20:51:33.917954 20561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37711 every 8 connection(s)
I20260929 20:51:33.919173 20435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-1/data/info.pb
I20260929 20:51:33.927398 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 20435
I20260929 20:51:33.927569 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-1/wal/instance
I20260929 20:51:33.928112 388 external_mini_cluster.cc:1428] Setting key 833e95cccac7c9065033f640e19f9882
I20260929 20:51:33.931166 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37961
--builtin_ntp_servers=127.0.97.20:45955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:51:33.932353 20284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:33.921842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39087 (local address 127.0.97.62:37961)
0929 20:51:33.922051 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:33.922058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:33.923470 (+ 1412us) server_negotiation.cc:407] Connection header received
0929 20:51:33.925240 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:33.925245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:33.925319 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:33.925416 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:33.926501 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:33.927353 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:33.929404 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:33.930026 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:33.930176 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:33.931649 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:33.931654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:33.931658 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:33.931675 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:33.931708 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:33.931720 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:33.931786 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:33.932029 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 20:51:33.932138 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:51:33.933677 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:33.922362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37961 (local address 127.0.97.2:39087)
0929 20:51:33.923326 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:33.923368 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:33.924959 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:33.925621 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:33.925629 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:33.925719 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:51:33.926338 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:33.926355 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:33.927514 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:33.927520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:33.929253 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:33.929266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:33.930275 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:33.930284 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:33.930568 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:33.931311 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:51:33.931315 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:51:33.931370 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:51:33.931466 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:51:33.931802 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:51:33.931813 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:51:33.931821 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 20:51:33.931995 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 20:51:33.932404 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":289,"threads_started":1}
I20260929 20:51:33.935001 20562 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37961
I20260929 20:51:33.935324 20562 heartbeater.cc:464] Registering TS with master...
I20260929 20:51:33.936180 20562 heartbeater.cc:511] Master 127.0.97.62:37961 requested a full tablet report, sending...
I20260929 20:51:33.937592 20246 ts_manager.cc:194] Registered new tserver with Master: 3f8101b8a1894e39b76746e9a8e70da9 (127.0.97.2:37711)
I20260929 20:51:33.938369 20246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:39087
W20260929 20:51:34.238193 20565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:34.238471 20565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:34.238593 20565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:34.245786 20565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:34.245875 20565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:34.246013 20565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 20:51:34.254911 20565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37961
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:34.256115 20565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:34.257776 20565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:34.270349 20570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:34.270294 20571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:34.270328 20573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:34.272291 20565 server_base.cc:1034] running on GCE node
I20260929 20:51:34.272907 20565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:34.273910 20565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:34.275097 20565 hybrid_clock.cc:648] HybridClock initialized: now 1790715094275045 us; error 39 us; skew 500 ppm
I20260929 20:51:34.275391 20565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:51:34.277551 20565 webserver.cc:460] Webserver started at http://127.0.97.3:34651/ using document root <none> and password file <none>
I20260929 20:51:34.278240 20565 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:34.278360 20565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:34.278707 20565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:34.281878 20565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-2/data/instance:
uuid: "6b1552c3d6c7420dafe34d65c4dfd6be"
format_stamp: "Formatted at 2026-09-29 20:51:34 on dist-test-slave-blv7"
server_key: "438d6f9c8e6643ec6265a0b28312bbb3"
server_key_iv: "ad29445732528d0ad4e2ab698d9d9822"
server_key_version: "encryptionkey@0"
I20260929 20:51:34.282666 20565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6b1552c3d6c7420dafe34d65c4dfd6be"
format_stamp: "Formatted at 2026-09-29 20:51:34 on dist-test-slave-blv7"
server_key: "438d6f9c8e6643ec6265a0b28312bbb3"
server_key_iv: "ad29445732528d0ad4e2ab698d9d9822"
server_key_version: "encryptionkey@0"
I20260929 20:51:34.287472 20565 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260929 20:51:34.290647 20579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:34.291885 20565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 20:51:34.292023 20565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-2/wal
uuid: "6b1552c3d6c7420dafe34d65c4dfd6be"
format_stamp: "Formatted at 2026-09-29 20:51:34 on dist-test-slave-blv7"
server_key: "438d6f9c8e6643ec6265a0b28312bbb3"
server_key_iv: "ad29445732528d0ad4e2ab698d9d9822"
server_key_version: "encryptionkey@0"
I20260929 20:51:34.292217 20565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:34.305737 20565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:34.306517 20565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:34.306768 20565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:34.307698 20565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:51:34.309088 20565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:51:34.309180 20565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:34.309288 20565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:51:34.309357 20565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:34.337862 20565 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37339
I20260929 20:51:34.337908 20691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37339 every 8 connection(s)
I20260929 20:51:34.339160 20565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-2/data/info.pb
I20260929 20:51:34.343672 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 20565
I20260929 20:51:34.343804 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-2/wal/instance
I20260929 20:51:34.344224 388 external_mini_cluster.cc:1428] Setting key 69a745b6a44c69c6484f8a98a9389199
I20260929 20:51:34.350754 20284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.341520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53099 (local address 127.0.97.62:37961)
0929 20:51:34.341722 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:34.341728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:34.343088 (+ 1360us) server_negotiation.cc:407] Connection header received
0929 20:51:34.344900 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:34.344906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:34.344986 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:34.345070 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:34.346103 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.346812 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.348385 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.348799 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.348907 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:34.350023 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:34.350028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:34.350031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:34.350048 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:34.350076 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:34.350088 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:34.350167 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:34.350359 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 20:51:34.350449 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:51:34.351693 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.342057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37961 (local address 127.0.97.3:53099)
0929 20:51:34.342946 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:34.342985 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:34.344655 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:34.345296 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:34.345305 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:34.345394 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:51:34.345939 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:34.345954 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.346944 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.346948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:34.348259 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:34.348271 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.348995 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.349001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:34.349210 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:34.349755 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:51:34.349757 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:51:34.349792 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:51:34.349859 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:51:34.350177 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:51:34.350187 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:51:34.350195 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 20:51:34.350336 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 20:51:34.350700 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":271,"threads_started":1}
I20260929 20:51:34.352768 20692 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37961
I20260929 20:51:34.352989 20692 heartbeater.cc:464] Registering TS with master...
I20260929 20:51:34.353636 20692 heartbeater.cc:511] Master 127.0.97.62:37961 requested a full tablet report, sending...
I20260929 20:51:34.354759 20246 ts_manager.cc:194] Registered new tserver with Master: 6b1552c3d6c7420dafe34d65c4dfd6be (127.0.97.3:37339)
I20260929 20:51:34.355525 20246 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:53099
I20260929 20:51:34.358358 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:51:34.372906 20284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.363275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34130 (local address 127.0.97.62:37961)
0929 20:51:34.363459 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:34.363465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:34.364197 (+ 732us) server_negotiation.cc:407] Connection header received
0929 20:51:34.364372 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:34.364378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:34.364517 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:34.364624 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:34.365275 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.366472 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.368393 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.370855 (+ 2462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.370984 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:34.372154 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:34.372159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:34.372162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:34.372182 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:34.372215 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:34.372227 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:34.372302 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:34.372456 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 20:51:34.372554 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:51:34.377296 20246 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:51:34.386143 20246 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:51:34.432159 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.400236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37339 (local address 127.0.0.1:35380)
0929 20:51:34.423939 (+ 23703us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:34.423955 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:34.424101 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:34.425215 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:34.425223 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:34.425256 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:34.425589 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:34.425599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.427534 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.427541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:34.429792 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:34.429804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.431201 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.431207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:34.431420 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:34.431477 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 20:51:34.431605 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 20:51:34.431807 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23576,"thread_start_us":25606,"threads_started":1}
I20260929 20:51:34.433316 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.400606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35380 (local address 127.0.97.3:37339)
0929 20:51:34.415568 (+ 14962us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:34.415575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:34.424085 (+ 8510us) server_negotiation.cc:407] Connection header received
0929 20:51:34.424343 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:34.424353 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:34.424632 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:34.424831 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:34.425788 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.427374 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.429990 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.430991 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.431163 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:34.431452 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:34.432765 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0929 20:51:34.432949 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14802,"thread_start_us":14656,"threads_started":1}
I20260929 20:51:34.433542 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.402309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36407 (local address 127.0.0.1:58382)
0929 20:51:34.426088 (+ 23779us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:34.426105 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:34.426245 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:34.426885 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:34.426889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:34.426915 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:34.427272 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:34.427284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.428719 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.428724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:34.430595 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:34.430609 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.432856 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.432860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:34.432979 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:34.433002 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:51:34.433076 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 20:51:34.433202 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23659,"mutex_wait_us":24,"thread_start_us":21346,"threads_started":1}
I20260929 20:51:34.433637 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.402842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58382 (local address 127.0.97.1:36407)
0929 20:51:34.423470 (+ 20628us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:34.423481 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:34.426259 (+ 2778us) server_negotiation.cc:407] Connection header received
0929 20:51:34.426405 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:34.426415 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:34.426705 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:34.426887 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:34.427464 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.428586 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.430790 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.432453 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.432572 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:34.432749 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:34.433156 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 20:51:34.433324 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20483,"thread_start_us":20377,"threads_started":1}
I20260929 20:51:34.433964 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.399869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37711 (local address 127.0.0.1:41066)
0929 20:51:34.413154 (+ 13285us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:34.413215 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:34.413400 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:34.427298 (+ 13898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:34.427303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:34.427331 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:34.427703 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:34.427716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.429542 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.429548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:34.431476 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:34.431487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.433184 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.433191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:34.433360 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:34.433390 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:51:34.433453 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 20:51:34.433570 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12937,"mutex_wait_us":1,"thread_start_us":12776,"threads_started":1}
I20260929 20:51:34.434964 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.400600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41066 (local address 127.0.97.2:37711)
0929 20:51:34.426706 (+ 26106us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:34.426716 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:34.426745 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:51:34.426843 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:34.426851 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:34.427104 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:34.427298 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:34.427897 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.429391 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.431683 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.432998 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.434182 (+ 1184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:34.434436 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:34.434504 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 20:51:34.434660 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25938,"thread_start_us":25821,"threads_started":1}
I20260929 20:51:34.437381 20627 tablet_service.cc:1467] Processing CreateTablet for tablet f10cfb6f19834492b9b8b8f776660f3c (DEFAULT_TABLE table=test-table [id=05e86dd487e5410b94ac321d03f6a2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:51:34.437536 20367 tablet_service.cc:1467] Processing CreateTablet for tablet f10cfb6f19834492b9b8b8f776660f3c (DEFAULT_TABLE table=test-table [id=05e86dd487e5410b94ac321d03f6a2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:51:34.438035 20497 tablet_service.cc:1467] Processing CreateTablet for tablet f10cfb6f19834492b9b8b8f776660f3c (DEFAULT_TABLE table=test-table [id=05e86dd487e5410b94ac321d03f6a2e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:51:34.438664 20627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f10cfb6f19834492b9b8b8f776660f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:34.438968 20367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f10cfb6f19834492b9b8b8f776660f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:34.439280 20497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f10cfb6f19834492b9b8b8f776660f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:34.453374 20710 tablet_bootstrap.cc:492] T f10cfb6f19834492b9b8b8f776660f3c P 6b1552c3d6c7420dafe34d65c4dfd6be: Bootstrap starting.
I20260929 20:51:34.455761 20711 tablet_bootstrap.cc:492] T f10cfb6f19834492b9b8b8f776660f3c P f8d4c00331ef4193ae8bbe8d224ff9d5: Bootstrap starting.
I20260929 20:51:34.457942 20710 tablet_bootstrap.cc:654] T f10cfb6f19834492b9b8b8f776660f3c P 6b1552c3d6c7420dafe34d65c4dfd6be: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:34.458434 20712 tablet_bootstrap.cc:492] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9: Bootstrap starting.
I20260929 20:51:34.459309 20711 tablet_bootstrap.cc:654] T f10cfb6f19834492b9b8b8f776660f3c P f8d4c00331ef4193ae8bbe8d224ff9d5: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:34.459340 20710 log.cc:824] T f10cfb6f19834492b9b8b8f776660f3c P 6b1552c3d6c7420dafe34d65c4dfd6be: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:34.460492 20711 log.cc:824] T f10cfb6f19834492b9b8b8f776660f3c P f8d4c00331ef4193ae8bbe8d224ff9d5: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:34.462175 20712 tablet_bootstrap.cc:654] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:34.462637 20710 tablet_bootstrap.cc:492] T f10cfb6f19834492b9b8b8f776660f3c P 6b1552c3d6c7420dafe34d65c4dfd6be: No bootstrap required, opened a new log
I20260929 20:51:34.462946 20710 ts_tablet_manager.cc:1397] T f10cfb6f19834492b9b8b8f776660f3c P 6b1552c3d6c7420dafe34d65c4dfd6be: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 20:51:34.463235 20712 log.cc:824] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:34.463534 20711 tablet_bootstrap.cc:492] T f10cfb6f19834492b9b8b8f776660f3c P f8d4c00331ef4193ae8bbe8d224ff9d5: No bootstrap required, opened a new log
I20260929 20:51:34.463797 20711 ts_tablet_manager.cc:1397] T f10cfb6f19834492b9b8b8f776660f3c P f8d4c00331ef4193ae8bbe8d224ff9d5: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 20:51:34.469709 20712 tablet_bootstrap.cc:492] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9: No bootstrap required, opened a new log
I20260929 20:51:34.470033 20712 ts_tablet_manager.cc:1397] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260929 20:51:34.473709 20710 raft_consensus.cc:348] T f10cfb6f19834492b9b8b8f776660f3c P 6b1552c3d6c7420dafe34d65c4dfd6be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8101b8a1894e39b76746e9a8e70da9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37711 } } peers { permanent_uuid: "6b1552c3d6c7420dafe34d65c4dfd6be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37339 } } peers { permanent_uuid: "f8d4c00331ef4193ae8bbe8d224ff9d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36407 } }
I20260929 20:51:34.474215 20710 raft_consensus.cc:374] T f10cfb6f19834492b9b8b8f776660f3c P 6b1552c3d6c7420dafe34d65c4dfd6be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:34.474325 20710 raft_consensus.cc:729] T f10cfb6f19834492b9b8b8f776660f3c P 6b1552c3d6c7420dafe34d65c4dfd6be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1552c3d6c7420dafe34d65c4dfd6be, State: Initialized, Role: FOLLOWER
I20260929 20:51:34.474548 20711 raft_consensus.cc:348] T f10cfb6f19834492b9b8b8f776660f3c P f8d4c00331ef4193ae8bbe8d224ff9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8101b8a1894e39b76746e9a8e70da9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37711 } } peers { permanent_uuid: "6b1552c3d6c7420dafe34d65c4dfd6be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37339 } } peers { permanent_uuid: "f8d4c00331ef4193ae8bbe8d224ff9d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36407 } }
I20260929 20:51:34.474968 20711 raft_consensus.cc:374] T f10cfb6f19834492b9b8b8f776660f3c P f8d4c00331ef4193ae8bbe8d224ff9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:34.475131 20711 raft_consensus.cc:729] T f10cfb6f19834492b9b8b8f776660f3c P f8d4c00331ef4193ae8bbe8d224ff9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8d4c00331ef4193ae8bbe8d224ff9d5, State: Initialized, Role: FOLLOWER
I20260929 20:51:34.474962 20710 consensus_queue.cc:260] T f10cfb6f19834492b9b8b8f776660f3c P 6b1552c3d6c7420dafe34d65c4dfd6be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8101b8a1894e39b76746e9a8e70da9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37711 } } peers { permanent_uuid: "6b1552c3d6c7420dafe34d65c4dfd6be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37339 } } peers { permanent_uuid: "f8d4c00331ef4193ae8bbe8d224ff9d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36407 } }
I20260929 20:51:34.475838 20711 consensus_queue.cc:260] T f10cfb6f19834492b9b8b8f776660f3c P f8d4c00331ef4193ae8bbe8d224ff9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8101b8a1894e39b76746e9a8e70da9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37711 } } peers { permanent_uuid: "6b1552c3d6c7420dafe34d65c4dfd6be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37339 } } peers { permanent_uuid: "f8d4c00331ef4193ae8bbe8d224ff9d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36407 } }
I20260929 20:51:34.476084 20692 heartbeater.cc:503] Master 127.0.97.62:37961 was elected leader, sending a full tablet report...
I20260929 20:51:34.476802 20710 ts_tablet_manager.cc:1428] T f10cfb6f19834492b9b8b8f776660f3c P 6b1552c3d6c7420dafe34d65c4dfd6be: Time spent starting tablet: real 0.014s user 0.005s sys 0.007s
I20260929 20:51:34.476943 20432 heartbeater.cc:503] Master 127.0.97.62:37961 was elected leader, sending a full tablet report...
I20260929 20:51:34.477545 20711 ts_tablet_manager.cc:1428] T f10cfb6f19834492b9b8b8f776660f3c P f8d4c00331ef4193ae8bbe8d224ff9d5: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260929 20:51:34.479761 20712 raft_consensus.cc:348] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8101b8a1894e39b76746e9a8e70da9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37711 } } peers { permanent_uuid: "6b1552c3d6c7420dafe34d65c4dfd6be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37339 } } peers { permanent_uuid: "f8d4c00331ef4193ae8bbe8d224ff9d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36407 } }
I20260929 20:51:34.480199 20712 raft_consensus.cc:374] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:34.480304 20712 raft_consensus.cc:729] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f8101b8a1894e39b76746e9a8e70da9, State: Initialized, Role: FOLLOWER
I20260929 20:51:34.480841 20712 consensus_queue.cc:260] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8101b8a1894e39b76746e9a8e70da9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37711 } } peers { permanent_uuid: "6b1552c3d6c7420dafe34d65c4dfd6be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37339 } } peers { permanent_uuid: "f8d4c00331ef4193ae8bbe8d224ff9d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36407 } }
I20260929 20:51:34.483753 20562 heartbeater.cc:503] Master 127.0.97.62:37961 was elected leader, sending a full tablet report...
I20260929 20:51:34.484261 20712 ts_tablet_manager.cc:1428] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260929 20:51:34.502391 20718 raft_consensus.cc:482] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:51:34.502694 20718 raft_consensus.cc:504] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8101b8a1894e39b76746e9a8e70da9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37711 } } peers { permanent_uuid: "6b1552c3d6c7420dafe34d65c4dfd6be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37339 } } peers { permanent_uuid: "f8d4c00331ef4193ae8bbe8d224ff9d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36407 } }
I20260929 20:51:34.504390 20718 leader_election.cc:288] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b1552c3d6c7420dafe34d65c4dfd6be (127.0.97.3:37339), f8d4c00331ef4193ae8bbe8d224ff9d5 (127.0.97.1:36407)
W20260929 20:51:34.511004 20433 tablet.cc:2369] T f10cfb6f19834492b9b8b8f776660f3c P f8d4c00331ef4193ae8bbe8d224ff9d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:51:34.511983 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.504972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59847 (local address 127.0.97.3:37339)
0929 20:51:34.505174 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:34.505180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:34.505838 (+ 658us) server_negotiation.cc:407] Connection header received
0929 20:51:34.506004 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:34.506009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:34.506081 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:34.506180 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:34.506895 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.507976 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.509921 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.510868 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.510989 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:34.511117 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:34.511603 (+ 486us) server_negotiation.cc:299] Negotiation successful
0929 20:51:34.511749 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 20:51:34.512661 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.504632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37339 (local address 127.0.97.2:59847)
0929 20:51:34.505731 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:34.505747 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:34.505873 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:34.506256 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:34.506260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:34.506308 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:34.506722 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:34.506732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.508100 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.508105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:34.509731 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:34.509745 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.511237 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.511243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:34.511418 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:34.511451 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:51:34.512160 (+ 709us) client_negotiation.cc:240] Negotiation successful
0929 20:51:34.512285 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"mutex_wait_us":325,"spinlock_wait_cycles":4992,"thread_start_us":594,"threads_started":1}
I20260929 20:51:34.512753 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.504632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36407 (local address 127.0.97.2:50403)
0929 20:51:34.505345 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:34.505369 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:51:34.505577 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:51:34.506094 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:51:34.506098 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:51:34.506123 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:51:34.506529 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:34.506539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.508017 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.508023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:34.510234 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:51:34.510248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.512016 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.512022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:51:34.512202 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:34.512233 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:51:34.512353 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 20:51:34.512464 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":92,"thread_start_us":375,"threads_started":1}
I20260929 20:51:34.512795 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.504954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50403 (local address 127.0.97.1:36407)
0929 20:51:34.505155 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:34.505161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:34.505455 (+ 294us) server_negotiation.cc:407] Connection header received
0929 20:51:34.505759 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:34.505765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:34.505852 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:34.505912 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:51:34.506721 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.507871 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.510419 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.511705 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.511819 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:34.511907 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:34.512401 (+ 494us) server_negotiation.cc:299] Negotiation successful
0929 20:51:34.512543 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:51:34.513768 20387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10cfb6f19834492b9b8b8f776660f3c" candidate_uuid: "3f8101b8a1894e39b76746e9a8e70da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8d4c00331ef4193ae8bbe8d224ff9d5" is_pre_election: true
I20260929 20:51:34.513760 20647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10cfb6f19834492b9b8b8f776660f3c" candidate_uuid: "3f8101b8a1894e39b76746e9a8e70da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1552c3d6c7420dafe34d65c4dfd6be" is_pre_election: true
I20260929 20:51:34.514276 20387 raft_consensus.cc:2454] T f10cfb6f19834492b9b8b8f776660f3c P f8d4c00331ef4193ae8bbe8d224ff9d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f8101b8a1894e39b76746e9a8e70da9 in term 0.
I20260929 20:51:34.514274 20647 raft_consensus.cc:2454] T f10cfb6f19834492b9b8b8f776660f3c P 6b1552c3d6c7420dafe34d65c4dfd6be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f8101b8a1894e39b76746e9a8e70da9 in term 0.
I20260929 20:51:34.514933 20453 leader_election.cc:302] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f8101b8a1894e39b76746e9a8e70da9, f8d4c00331ef4193ae8bbe8d224ff9d5; no voters:
I20260929 20:51:34.515388 20718 raft_consensus.cc:2781] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:51:34.515525 20718 raft_consensus.cc:482] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:51:34.515627 20718 raft_consensus.cc:3037] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:34.517813 20718 raft_consensus.cc:504] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8101b8a1894e39b76746e9a8e70da9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37711 } } peers { permanent_uuid: "6b1552c3d6c7420dafe34d65c4dfd6be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37339 } } peers { permanent_uuid: "f8d4c00331ef4193ae8bbe8d224ff9d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36407 } }
I20260929 20:51:34.518396 20718 leader_election.cc:288] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9 [CANDIDATE]: Term 1 election: Requested vote from peers 6b1552c3d6c7420dafe34d65c4dfd6be (127.0.97.3:37339), f8d4c00331ef4193ae8bbe8d224ff9d5 (127.0.97.1:36407)
I20260929 20:51:34.518819 20647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10cfb6f19834492b9b8b8f776660f3c" candidate_uuid: "3f8101b8a1894e39b76746e9a8e70da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1552c3d6c7420dafe34d65c4dfd6be"
I20260929 20:51:34.518906 20387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10cfb6f19834492b9b8b8f776660f3c" candidate_uuid: "3f8101b8a1894e39b76746e9a8e70da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8d4c00331ef4193ae8bbe8d224ff9d5"
I20260929 20:51:34.519048 20647 raft_consensus.cc:3037] T f10cfb6f19834492b9b8b8f776660f3c P 6b1552c3d6c7420dafe34d65c4dfd6be [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:34.519114 20387 raft_consensus.cc:3037] T f10cfb6f19834492b9b8b8f776660f3c P f8d4c00331ef4193ae8bbe8d224ff9d5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:34.522260 20647 raft_consensus.cc:2454] T f10cfb6f19834492b9b8b8f776660f3c P 6b1552c3d6c7420dafe34d65c4dfd6be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f8101b8a1894e39b76746e9a8e70da9 in term 1.
I20260929 20:51:34.522336 20387 raft_consensus.cc:2454] T f10cfb6f19834492b9b8b8f776660f3c P f8d4c00331ef4193ae8bbe8d224ff9d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f8101b8a1894e39b76746e9a8e70da9 in term 1.
I20260929 20:51:34.522742 20450 leader_election.cc:302] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3f8101b8a1894e39b76746e9a8e70da9, 6b1552c3d6c7420dafe34d65c4dfd6be, f8d4c00331ef4193ae8bbe8d224ff9d5; no voters:
I20260929 20:51:34.523108 20718 raft_consensus.cc:2781] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:34.523551 20718 raft_consensus.cc:686] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9 [term 1 LEADER]: Becoming Leader. State: Replica: 3f8101b8a1894e39b76746e9a8e70da9, State: Running, Role: LEADER
I20260929 20:51:34.524046 20718 consensus_queue.cc:237] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8101b8a1894e39b76746e9a8e70da9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37711 } } peers { permanent_uuid: "6b1552c3d6c7420dafe34d65c4dfd6be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37339 } } peers { permanent_uuid: "f8d4c00331ef4193ae8bbe8d224ff9d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36407 } }
I20260929 20:51:34.528357 20245 catalog_manager.cc:5495] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f8101b8a1894e39b76746e9a8e70da9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3f8101b8a1894e39b76746e9a8e70da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8101b8a1894e39b76746e9a8e70da9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b1552c3d6c7420dafe34d65c4dfd6be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8d4c00331ef4193ae8bbe8d224ff9d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36407 } health_report { overall_health: UNKNOWN } } }
I20260929 20:51:34.570837 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.565964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.0.97.2:37711)
0929 20:51:34.566162 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:34.566166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:34.566181 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:51:34.566263 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:34.566266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:34.566365 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:34.566497 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:51:34.566795 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.567570 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.568901 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.569369 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.569477 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:34.569862 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:51:34.570432 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:51:34.570547 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:34.570583 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 20:51:34.570644 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:51:34.578538 20647 raft_consensus.cc:1261] T f10cfb6f19834492b9b8b8f776660f3c P 6b1552c3d6c7420dafe34d65c4dfd6be [term 1 FOLLOWER]: Refusing update from remote peer 3f8101b8a1894e39b76746e9a8e70da9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:51:34.578559 20387 raft_consensus.cc:1261] T f10cfb6f19834492b9b8b8f776660f3c P f8d4c00331ef4193ae8bbe8d224ff9d5 [term 1 FOLLOWER]: Refusing update from remote peer 3f8101b8a1894e39b76746e9a8e70da9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:51:34.579483 20721 consensus_queue.cc:1035] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b1552c3d6c7420dafe34d65c4dfd6be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:51:34.579809 20718 consensus_queue.cc:1035] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8d4c00331ef4193ae8bbe8d224ff9d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 20:51:34.590633 20693 tablet.cc:2369] T f10cfb6f19834492b9b8b8f776660f3c P 6b1552c3d6c7420dafe34d65c4dfd6be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:51:34.602449 20246 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:51:34.602763 20246 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:51:34.606338 20246 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 087a26dbe3484afea52a7c10f95c043b: Sending DeleteTablet for 3 replicas of tablet f10cfb6f19834492b9b8b8f776660f3c
I20260929 20:51:34.607534 20497 tablet_service.cc:1514] Processing DeleteTablet for tablet f10cfb6f19834492b9b8b8f776660f3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:51:34 UTC) from {username='slave'} at 127.0.0.1:41066
I20260929 20:51:34.607582 20627 tablet_service.cc:1514] Processing DeleteTablet for tablet f10cfb6f19834492b9b8b8f776660f3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:51:34 UTC) from {username='slave'} at 127.0.0.1:35380
I20260929 20:51:34.608009 20367 tablet_service.cc:1514] Processing DeleteTablet for tablet f10cfb6f19834492b9b8b8f776660f3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:51:34 UTC) from {username='slave'} at 127.0.0.1:58382
I20260929 20:51:34.608860 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 20305
I20260929 20:51:34.615507 20734 tablet_replica.cc:330] T f10cfb6f19834492b9b8b8f776660f3c P 6b1552c3d6c7420dafe34d65c4dfd6be: stopping tablet replica
I20260929 20:51:34.615738 20735 tablet_replica.cc:330] T f10cfb6f19834492b9b8b8f776660f3c P f8d4c00331ef4193ae8bbe8d224ff9d5: stopping tablet replica
I20260929 20:51:34.615959 20734 raft_consensus.cc:2229] T f10cfb6f19834492b9b8b8f776660f3c P 6b1552c3d6c7420dafe34d65c4dfd6be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:51:34.616185 20735 raft_consensus.cc:2229] T f10cfb6f19834492b9b8b8f776660f3c P f8d4c00331ef4193ae8bbe8d224ff9d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:51:34.616509 20734 raft_consensus.cc:2258] T f10cfb6f19834492b9b8b8f776660f3c P 6b1552c3d6c7420dafe34d65c4dfd6be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:51:34.616524 20733 tablet_replica.cc:330] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9: stopping tablet replica
I20260929 20:51:34.616811 20735 raft_consensus.cc:2258] T f10cfb6f19834492b9b8b8f776660f3c P f8d4c00331ef4193ae8bbe8d224ff9d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:51:34.617094 20733 raft_consensus.cc:2229] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:51:34.617758 20733 raft_consensus.cc:2258] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:51:34.619596 20735 ts_tablet_manager.cc:1905] T f10cfb6f19834492b9b8b8f776660f3c P f8d4c00331ef4193ae8bbe8d224ff9d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:51:34.619689 20734 ts_tablet_manager.cc:1905] T f10cfb6f19834492b9b8b8f776660f3c P 6b1552c3d6c7420dafe34d65c4dfd6be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:51:34.620894 20733 ts_tablet_manager.cc:1905] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:51:34.625872 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.610158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58394 (local address 127.0.97.1:36407)
0929 20:51:34.610334 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:34.610341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:34.615957 (+ 5616us) server_negotiation.cc:407] Connection header received
0929 20:51:34.616131 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:34.616137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:34.616272 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:34.616372 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:34.617107 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.618267 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.621507 (+ 3240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.622165 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.622289 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:34.623719 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:34.623730 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:34.623746 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:34.624479 (+ 733us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:34.624548 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:34.624573 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:34.624691 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:34.625239 (+ 548us) server_negotiation.cc:299] Negotiation successful
0929 20:51:34.625351 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":57}
I20260929 20:51:34.626799 20427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:34.626909 20734 ts_tablet_manager.cc:1918] T f10cfb6f19834492b9b8b8f776660f3c P 6b1552c3d6c7420dafe34d65c4dfd6be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:51:34.627121 20734 log.cc:1196] T f10cfb6f19834492b9b8b8f776660f3c P 6b1552c3d6c7420dafe34d65c4dfd6be: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-2/wal/wals/f10cfb6f19834492b9b8b8f776660f3c
I20260929 20:51:34.627645 20734 ts_tablet_manager.cc:1939] T f10cfb6f19834492b9b8b8f776660f3c P 6b1552c3d6c7420dafe34d65c4dfd6be: Deleting consensus metadata
I20260929 20:51:34.629657 20230 catalog_manager.cc:4841] TS 6b1552c3d6c7420dafe34d65c4dfd6be (127.0.97.3:37339): tablet f10cfb6f19834492b9b8b8f776660f3c (table test-table [id=05e86dd487e5410b94ac321d03f6a2e7]) successfully deleted
I20260929 20:51:34.636832 20733 ts_tablet_manager.cc:1918] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:51:34.637037 20733 log.cc:1196] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-1/wal/wals/f10cfb6f19834492b9b8b8f776660f3c
I20260929 20:51:34.637640 20733 ts_tablet_manager.cc:1939] T f10cfb6f19834492b9b8b8f776660f3c P 3f8101b8a1894e39b76746e9a8e70da9: Deleting consensus metadata
I20260929 20:51:34.639632 20231 catalog_manager.cc:4841] TS 3f8101b8a1894e39b76746e9a8e70da9 (127.0.97.2:37711): tablet f10cfb6f19834492b9b8b8f776660f3c (table test-table [id=05e86dd487e5410b94ac321d03f6a2e7]) successfully deleted
I20260929 20:51:34.721185 20735 ts_tablet_manager.cc:1918] T f10cfb6f19834492b9b8b8f776660f3c P f8d4c00331ef4193ae8bbe8d224ff9d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:51:34.721432 20735 log.cc:1196] T f10cfb6f19834492b9b8b8f776660f3c P f8d4c00331ef4193ae8bbe8d224ff9d5: Deleting WAL directory at /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790714786636870-388-0/minicluster-data/ts-0/wal/wals/f10cfb6f19834492b9b8b8f776660f3c
I20260929 20:51:34.721899 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 20305
I20260929 20:51:34.736171 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 20435
W20260929 20:51:34.736056 20233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:36407 (error 108)
I20260929 20:51:34.738157 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.737611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36407 (local address 127.0.0.1:58404)
0929 20:51:34.737873 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:34.737988 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260929 20:51:34.738444 20233 catalog_manager.cc:4584] TS f8d4c00331ef4193ae8bbe8d224ff9d5 (127.0.97.1:36407): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f10cfb6f19834492b9b8b8f776660f3c: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36407: connect: Connection refused (error 111)
I20260929 20:51:34.743482 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.737242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41096 (local address 127.0.97.2:37711)
0929 20:51:34.737418 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:34.737425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:34.737440 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:51:34.737512 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:34.737525 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:34.737631 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:34.737731 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:34.738254 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.739264 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.740629 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.741075 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.741180 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:34.742127 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:34.742136 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:34.742149 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:34.742738 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:34.742784 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:34.742811 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:34.742912 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:34.743072 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 20:51:34.743167 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:51:34.744211 20557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:34.792465 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.791963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36407 (local address 127.0.0.1:58414)
0929 20:51:34.792210 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:34.792306 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 20:51:34.793555 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.793193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36407 (local address 127.0.0.1:58426)
0929 20:51:34.793346 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:34.793437 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 20:51:34.852998 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 20435
I20260929 20:51:34.854036 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.853488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36407 (local address 127.0.0.1:58434)
0929 20:51:34.853716 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:34.853828 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 20:51:34.855238 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.854835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36407 (local address 127.0.0.1:58446)
0929 20:51:34.854990 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:34.855104 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 20:51:34.867820 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 20565
I20260929 20:51:34.874375 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.868503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35388 (local address 127.0.97.3:37339)
0929 20:51:34.868673 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:34.868679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:34.868757 (+ 78us) server_negotiation.cc:407] Connection header received
0929 20:51:34.868942 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:34.868946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:34.869026 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:34.869144 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:34.869541 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.870257 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.871530 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:34.871909 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:34.872015 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:34.873030 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:34.873039 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:34.873053 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:34.873701 (+ 648us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:34.873749 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:34.873769 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:34.873865 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:34.873992 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 20:51:34.874096 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:51:34.874996 20687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:34.935760 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.935318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36407 (local address 127.0.0.1:58448)
0929 20:51:34.935509 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:34.935605 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 20:51:34.936843 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:34.936448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36407 (local address 127.0.0.1:58454)
0929 20:51:34.936661 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 20:51:34.936733 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260929 20:51:34.961476 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 20565
I20260929 20:51:34.974720 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 20215
I20260929 20:51:34.975739 20276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:35.081406 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 20215
2026-09-29T20:51:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2567 ms)
[----------] 6 tests from EncryptionPolicyTest (20161 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:51:35 dist-test-slave-blv7 krb5kdc[20744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:51:35 dist-test-slave-blv7 krb5kdc[20744](info): set up 2 sockets
Sep 29 20:51:35 dist-test-slave-blv7 krb5kdc[20744](info): commencing operation
krb5kdc: starting...
W20260929 20:51:36.095132 388 mini_kdc.cc:121] Time spent starting KDC: real 0.977s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:51:36 dist-test-slave-blv7 krb5kdc[20744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:51:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:51:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:51:36.322485 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37979
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:36.637688 20757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:36.637976 20757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:36.638113 20757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:36.645143 20757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:51:36.645257 20757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:36.645334 20757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:36.645409 20757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:51:36.645468 20757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:51:36.655133 20757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37979
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:36.656364 20757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:36.657579 20757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:36.668556 20762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:36.668586 20765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:36.668510 20763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:36.669234 20757 server_base.cc:1034] running on GCE node
I20260929 20:51:36.669961 20757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:36.671350 20757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:36.672547 20757 hybrid_clock.cc:648] HybridClock initialized: now 1790715096672497 us; error 37 us; skew 500 ppm
I20260929 20:51:36.672889 20757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:36 dist-test-slave-blv7 krb5kdc[20744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:36.675767 20757 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:51:36.677115 20757 webserver.cc:460] Webserver started at http://127.0.0.1:38961/ using document root <none> and password file <none>
I20260929 20:51:36.677752 20757 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:36.677865 20757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:36.678136 20757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:36.681069 20757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "04fa980f9bb24a339b5fc005a4d9ab82"
format_stamp: "Formatted at 2026-09-29 20:51:36 on dist-test-slave-blv7"
server_key: "31840406d357470924ae85b4389d07fb"
server_key_iv: "0c4491335cb0bdf857b8439a452d7caf"
server_key_version: "encryptionkey@0"
I20260929 20:51:36.681777 20757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "04fa980f9bb24a339b5fc005a4d9ab82"
format_stamp: "Formatted at 2026-09-29 20:51:36 on dist-test-slave-blv7"
server_key: "31840406d357470924ae85b4389d07fb"
server_key_iv: "0c4491335cb0bdf857b8439a452d7caf"
server_key_version: "encryptionkey@0"
I20260929 20:51:36.686645 20757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 20:51:36.690037 20772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:36.691382 20757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:51:36.691530 20757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "04fa980f9bb24a339b5fc005a4d9ab82"
format_stamp: "Formatted at 2026-09-29 20:51:36 on dist-test-slave-blv7"
server_key: "31840406d357470924ae85b4389d07fb"
server_key_iv: "0c4491335cb0bdf857b8439a452d7caf"
server_key_version: "encryptionkey@0"
I20260929 20:51:36.691712 20757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:36.726768 20757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:36.730877 20757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:36.731187 20757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:36.748000 20757 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37979
I20260929 20:51:36.748019 20823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37979 every 8 connection(s)
I20260929 20:51:36.749339 20757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:51:36.753037 20824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:36.754276 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 20757
I20260929 20:51:36.754483 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:51:36.755267 388 external_mini_cluster.cc:1428] Setting key 1bae2e2cf97d6d230e84af9e12b72dd1
I20260929 20:51:36.767410 20824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82: Bootstrap starting.
Sep 29 20:51:36 dist-test-slave-blv7 krb5kdc[20744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 20:51:36.771330 20824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:36.772445 20824 log.cc:824] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:36.775844 20824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82: No bootstrap required, opened a new log
I20260929 20:51:36.779366 20827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:36.757204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49175 (local address 127.0.0.1:37979)
0929 20:51:36.757974 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:36.757985 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:36.758028 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:51:36.759677 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:36.759700 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:36.760114 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:36.760453 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:36.762150 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:36.763535 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:36.765360 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:36.766076 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:36.766214 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:36.771089 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:36.771101 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:36.771117 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:36.771157 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:36.774379 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:36.775350 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:36.775357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:36.775363 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:36.775526 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:36.776066 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:36.776070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:36.776072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:36.776603 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:36.776848 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:36.777429 (+ 581us) server_negotiation.cc:299] Negotiation successful
0929 20:51:36.777801 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":363,"threads_started":1}
I20260929 20:51:36.783761 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:51:36.785549 20824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fa980f9bb24a339b5fc005a4d9ab82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37979 } }
I20260929 20:51:36.786001 20824 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:36.786106 20824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04fa980f9bb24a339b5fc005a4d9ab82, State: Initialized, Role: FOLLOWER
I20260929 20:51:36.786806 20824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fa980f9bb24a339b5fc005a4d9ab82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37979 } }
I20260929 20:51:36.787196 20824 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:51:36.787349 20824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:51:36.787545 20824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:36.790431 20824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fa980f9bb24a339b5fc005a4d9ab82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37979 } }
I20260929 20:51:36.790913 20824 leader_election.cc:302] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04fa980f9bb24a339b5fc005a4d9ab82; no voters:
I20260929 20:51:36.791478 20824 leader_election.cc:288] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:51:36.791627 20834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:36.792681 20834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82 [term 1 LEADER]: Becoming Leader. State: Replica: 04fa980f9bb24a339b5fc005a4d9ab82, State: Running, Role: LEADER
I20260929 20:51:36.793272 20834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fa980f9bb24a339b5fc005a4d9ab82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37979 } }
I20260929 20:51:36.793630 20824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:51:36.795979 20836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04fa980f9bb24a339b5fc005a4d9ab82. Latest consensus state: current_term: 1 leader_uuid: "04fa980f9bb24a339b5fc005a4d9ab82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fa980f9bb24a339b5fc005a4d9ab82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37979 } } }
I20260929 20:51:36.796356 20836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:36.796367 20835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04fa980f9bb24a339b5fc005a4d9ab82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fa980f9bb24a339b5fc005a4d9ab82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37979 } } }
I20260929 20:51:36.797024 20835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:36.797277 20841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:51:36.804268 20841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:51:36.806690 20827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:36.789359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38623 (local address 127.0.0.1:37979)
0929 20:51:36.789547 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:36.789554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:36.789921 (+ 367us) server_negotiation.cc:407] Connection header received
0929 20:51:36.790053 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:36.790058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:36.790133 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:36.790243 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:36.791761 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:36.792917 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:36.794677 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:36.795455 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:36.795601 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:36.798451 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:36.798457 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:36.798462 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:36.798500 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:36.802362 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:36.803388 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:36.803393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:36.803396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:36.803567 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:36.804131 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:36.804135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:36.804139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:36.804565 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:36.805924 (+ 1359us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:36.806104 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 20:51:36.806299 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:51:36.817099 20841 catalog_manager.cc:1347] Generated new cluster ID: 1f44605d51644cb18aa77b2426fb8adb
I20260929 20:51:36.817214 20841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:51:36.833837 20841 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:51:36.835668 20841 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:51:36.847731 20841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04fa980f9bb24a339b5fc005a4d9ab82: Generated new TSK 0
I20260929 20:51:36.848752 20841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:51:36.861699 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 20757
I20260929 20:51:36.863125 20819 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:36.954396 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 20757
W20260929 20:51:36.968079 20742 connection.cc:535] client connection to 127.0.0.1:37979 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37979): Connection reset by peer (error 104)
2026-09-29T20:51:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1869 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:51:37 dist-test-slave-blv7 krb5kdc[20856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:51:37 dist-test-slave-blv7 krb5kdc[20856](info): set up 2 sockets
Sep 29 20:51:37 dist-test-slave-blv7 krb5kdc[20856](info): commencing operation
krb5kdc: starting...
W20260929 20:51:37.958961 388 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:51:38 dist-test-slave-blv7 krb5kdc[20856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:51:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:51:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:51:38.186347 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38517
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:38.506242 20869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:38.506531 20869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:38.506667 20869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:38.513818 20869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:51:38.513943 20869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:38.514029 20869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:38.514112 20869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:51:38.514169 20869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:51:38.523993 20869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38517
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:38.525225 20869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:38.526485 20869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:38.537267 20875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:38.537225 20874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:38.537452 20877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:38.538087 20869 server_base.cc:1034] running on GCE node
I20260929 20:51:38.538801 20869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:38.540115 20869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:38.541326 20869 hybrid_clock.cc:648] HybridClock initialized: now 1790715098541271 us; error 48 us; skew 500 ppm
I20260929 20:51:38.541646 20869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:38 dist-test-slave-blv7 krb5kdc[20856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:38.544433 20869 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:51:38.545768 20869 webserver.cc:460] Webserver started at http://127.0.0.1:34007/ using document root <none> and password file <none>
I20260929 20:51:38.546391 20869 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:38.546509 20869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:38.546779 20869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:38.549659 20869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "6c8e10c6d45b4d58a412a531cb93e59c"
format_stamp: "Formatted at 2026-09-29 20:51:38 on dist-test-slave-blv7"
server_key: "c03838f7e8721aa2d9f6f2a10a8aff91"
server_key_iv: "0ac23804f21a9bcbc8741921d9251dc0"
server_key_version: "encryptionkey@0"
I20260929 20:51:38.550354 20869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "6c8e10c6d45b4d58a412a531cb93e59c"
format_stamp: "Formatted at 2026-09-29 20:51:38 on dist-test-slave-blv7"
server_key: "c03838f7e8721aa2d9f6f2a10a8aff91"
server_key_iv: "0ac23804f21a9bcbc8741921d9251dc0"
server_key_version: "encryptionkey@0"
I20260929 20:51:38.555119 20869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260929 20:51:38.558470 20884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:38.559795 20869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 20:51:38.559940 20869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "6c8e10c6d45b4d58a412a531cb93e59c"
format_stamp: "Formatted at 2026-09-29 20:51:38 on dist-test-slave-blv7"
server_key: "c03838f7e8721aa2d9f6f2a10a8aff91"
server_key_iv: "0ac23804f21a9bcbc8741921d9251dc0"
server_key_version: "encryptionkey@0"
I20260929 20:51:38.560102 20869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:38.571197 20869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:38.574563 20869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:38.574791 20869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:38.591522 20869 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38517
I20260929 20:51:38.591540 20935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38517 every 8 connection(s)
I20260929 20:51:38.592952 20869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:51:38.596638 20936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:38.599200 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 20869
I20260929 20:51:38.599390 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:51:38.599922 388 external_mini_cluster.cc:1428] Setting key ea1212ddc2583088f3dcd88b20a0d5bb
I20260929 20:51:38.607949 20936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c: Bootstrap starting.
I20260929 20:51:38.610996 20936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:38.612097 20936 log.cc:824] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:38.612830 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:38.601731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48469 (local address 127.0.0.1:38517)
0929 20:51:38.602625 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:38.602637 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:38.602676 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:51:38.604381 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:38.604408 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:38.604815 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:38.605214 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:38.605749 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:38.607100 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:38.608422 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:38.609090 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:38.609258 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:38.610467 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:38.610474 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:38.610489 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:38.611076 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:38.611164 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:38.611191 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:38.611339 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:38.611592 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 20:51:38.611743 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":316,"threads_started":1}
I20260929 20:51:38.615059 20936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c: No bootstrap required, opened a new log
I20260929 20:51:38.616523 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:51:38.622852 20936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8e10c6d45b4d58a412a531cb93e59c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38517 } }
I20260929 20:51:38.623328 20936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:38.623431 20936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c8e10c6d45b4d58a412a531cb93e59c, State: Initialized, Role: FOLLOWER
I20260929 20:51:38.624116 20936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8e10c6d45b4d58a412a531cb93e59c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38517 } }
I20260929 20:51:38.624392 20936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:51:38.624537 20936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:51:38.624706 20936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:38.627197 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:38.620484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54473 (local address 127.0.0.1:38517)
0929 20:51:38.620764 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:38.620778 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:38.621080 (+ 302us) server_negotiation.cc:407] Connection header received
0929 20:51:38.621254 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:38.621261 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:38.621323 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:38.621415 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:38.621941 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:38.622663 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:38.624485 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:38.625043 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:38.625141 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:38.626323 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:38.626333 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:38.626336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:38.626359 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:38.626416 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:38.626434 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:38.626534 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:38.626783 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 20:51:38.626847 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:51:38.627538 20936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8e10c6d45b4d58a412a531cb93e59c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38517 } }
I20260929 20:51:38.628055 20936 leader_election.cc:302] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c8e10c6d45b4d58a412a531cb93e59c; no voters:
I20260929 20:51:38.628633 20936 leader_election.cc:288] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:51:38.628686 20946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:38.629297 20946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c [term 1 LEADER]: Becoming Leader. State: Replica: 6c8e10c6d45b4d58a412a531cb93e59c, State: Running, Role: LEADER
I20260929 20:51:38.629822 20946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8e10c6d45b4d58a412a531cb93e59c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38517 } }
I20260929 20:51:38.630517 20936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:51:38.632351 20947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c8e10c6d45b4d58a412a531cb93e59c. Latest consensus state: current_term: 1 leader_uuid: "6c8e10c6d45b4d58a412a531cb93e59c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8e10c6d45b4d58a412a531cb93e59c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38517 } } }
I20260929 20:51:38.632732 20947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:38.634181 20951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:51:38.634130 20948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c8e10c6d45b4d58a412a531cb93e59c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c8e10c6d45b4d58a412a531cb93e59c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38517 } } }
I20260929 20:51:38.634423 20948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:38.639273 20951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:51:38.648124 20951 catalog_manager.cc:1347] Generated new cluster ID: 07fc3238b94e4dc6906636658b2ae7a2
I20260929 20:51:38.648269 20951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:51:38.662569 20951 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:51:38.664022 20951 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:51:38.677196 20951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c8e10c6d45b4d58a412a531cb93e59c: Generated new TSK 0
I20260929 20:51:38.678009 20951 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:51:38.684880 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 20869
W20260929 20:51:38.686466 20931 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48469
W20260929 20:51:38.687245 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 20869: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:51:38.687445 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 20869
W20260929 20:51:38.700608 20854 connection.cc:535] client connection to 127.0.0.1:38517 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38517): Connection reset by peer (error 104)
2026-09-29T20:51:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1732 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:51:38 dist-test-slave-blv7 krb5kdc[20967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:51:38 dist-test-slave-blv7 krb5kdc[20967](info): set up 2 sockets
Sep 29 20:51:38 dist-test-slave-blv7 krb5kdc[20967](info): commencing operation
krb5kdc: starting...
W20260929 20:51:39.694897 388 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:51:39 dist-test-slave-blv7 krb5kdc[20967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:51:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:51:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:51:39.924310 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45817
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:40.233889 20980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:40.234179 20980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:40.234274 20980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:40.241410 20980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:51:40.241521 20980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:40.241573 20980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:40.241631 20980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:51:40.241673 20980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:51:40.251340 20980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45817
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:40.252544 20980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:40.253787 20980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:40.265760 20986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:40.265779 20988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:40.265815 20985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:40.266590 20980 server_base.cc:1034] running on GCE node
I20260929 20:51:40.267416 20980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:40.268818 20980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:40.270033 20980 hybrid_clock.cc:648] HybridClock initialized: now 1790715100269967 us; error 50 us; skew 500 ppm
I20260929 20:51:40.270372 20980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:40 dist-test-slave-blv7 krb5kdc[20967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:40.278812 20980 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:51:40.280370 20980 webserver.cc:460] Webserver started at http://127.0.0.1:39867/ using document root <none> and password file <none>
I20260929 20:51:40.281204 20980 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:40.281332 20980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:40.281678 20980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:40.284785 20980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "e4db743f85e5446a8e5a5a12f89ab501"
format_stamp: "Formatted at 2026-09-29 20:51:40 on dist-test-slave-blv7"
server_key: "317de26c97354b128e9725bc9614b9bf"
server_key_iv: "de1fcbe722dfa1980e611152cdb2ea0a"
server_key_version: "encryptionkey@0"
I20260929 20:51:40.285511 20980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "e4db743f85e5446a8e5a5a12f89ab501"
format_stamp: "Formatted at 2026-09-29 20:51:40 on dist-test-slave-blv7"
server_key: "317de26c97354b128e9725bc9614b9bf"
server_key_iv: "de1fcbe722dfa1980e611152cdb2ea0a"
server_key_version: "encryptionkey@0"
I20260929 20:51:40.290512 20980 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:51:40.294032 20995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:40.295379 20980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:51:40.295523 20980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "e4db743f85e5446a8e5a5a12f89ab501"
format_stamp: "Formatted at 2026-09-29 20:51:40 on dist-test-slave-blv7"
server_key: "317de26c97354b128e9725bc9614b9bf"
server_key_iv: "de1fcbe722dfa1980e611152cdb2ea0a"
server_key_version: "encryptionkey@0"
I20260929 20:51:40.295738 20980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:40.301301 20980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:40.305437 20980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:40.305711 20980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:40.321660 20980 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45817
I20260929 20:51:40.321707 21046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45817 every 8 connection(s)
I20260929 20:51:40.322962 20980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:51:40.324807 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 20980
I20260929 20:51:40.324965 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:51:40.325377 388 external_mini_cluster.cc:1428] Setting key 1b57c846bd1f6138a4bd0f96bc3e9395
I20260929 20:51:40.326853 21047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:40.333644 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:40.326990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41781 (local address 127.0.0.1:45817)
0929 20:51:40.327801 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:40.327815 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:40.327862 (+ 47us) server_negotiation.cc:407] Connection header received
0929 20:51:40.329886 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:40.329914 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:40.330355 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:40.330802 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:40.331785 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:40.331793 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:40.331814 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:40.331830 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:40.331910 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:40.331940 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:40.332153 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:40.332277 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 20:51:40.332472 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":320,"threads_started":1}
I20260929 20:51:40.337033 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:51:40.340534 21047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501: Bootstrap starting.
I20260929 20:51:40.344327 21047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:40.344498 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:40.342046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37959 (local address 127.0.0.1:45817)
0929 20:51:40.342225 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:40.342231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:40.342672 (+ 441us) server_negotiation.cc:407] Connection header received
0929 20:51:40.342825 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:40.342828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:40.342874 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:40.342949 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:40.343847 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:40.343851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:40.343854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:40.343868 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:40.343905 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:40.343920 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:40.344000 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:40.344132 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 20:51:40.344175 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 20:51:40.345436 21047 log.cc:824] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:40.348038 21047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501: No bootstrap required, opened a new log
I20260929 20:51:40.355240 21047 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4db743f85e5446a8e5a5a12f89ab501" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45817 } }
I20260929 20:51:40.355593 21047 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:40.355683 21047 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4db743f85e5446a8e5a5a12f89ab501, State: Initialized, Role: FOLLOWER
I20260929 20:51:40.356223 21047 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4db743f85e5446a8e5a5a12f89ab501" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45817 } }
I20260929 20:51:40.356432 21047 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:51:40.356556 21047 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:51:40.356698 21047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:40.359011 21047 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4db743f85e5446a8e5a5a12f89ab501" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45817 } }
I20260929 20:51:40.359515 21047 leader_election.cc:302] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4db743f85e5446a8e5a5a12f89ab501; no voters:
I20260929 20:51:40.360051 21047 leader_election.cc:288] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:51:40.360172 21057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:40.360807 21057 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501 [term 1 LEADER]: Becoming Leader. State: Replica: e4db743f85e5446a8e5a5a12f89ab501, State: Running, Role: LEADER
I20260929 20:51:40.361312 21057 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4db743f85e5446a8e5a5a12f89ab501" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45817 } }
I20260929 20:51:40.361943 21047 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:51:40.363867 21059 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4db743f85e5446a8e5a5a12f89ab501. Latest consensus state: current_term: 1 leader_uuid: "e4db743f85e5446a8e5a5a12f89ab501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4db743f85e5446a8e5a5a12f89ab501" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45817 } } }
I20260929 20:51:40.364241 21059 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:40.364898 21058 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4db743f85e5446a8e5a5a12f89ab501" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4db743f85e5446a8e5a5a12f89ab501" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45817 } } }
I20260929 20:51:40.365185 21058 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:40.366302 21064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:51:40.371321 21064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:51:40.380424 21064 catalog_manager.cc:1347] Generated new cluster ID: 8597f528c4e349d4b17c68b3aad33684
I20260929 20:51:40.380527 21064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:51:40.399068 21064 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:51:40.399621 21064 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:51:40.407898 21064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4db743f85e5446a8e5a5a12f89ab501: Generated new TSK 0
I20260929 20:51:40.408983 21064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:51:40.417150 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 20980
W20260929 20:51:40.418648 21042 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41781
W20260929 20:51:40.419291 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 20980: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:51:40.419445 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 20980
2026-09-29T20:51:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1733 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:51:40 dist-test-slave-blv7 krb5kdc[21078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:51:40 dist-test-slave-blv7 krb5kdc[21078](info): set up 2 sockets
Sep 29 20:51:40 dist-test-slave-blv7 krb5kdc[21078](info): commencing operation
krb5kdc: starting...
W20260929 20:51:41.423097 388 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:51:41 dist-test-slave-blv7 krb5kdc[21078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:51:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:51:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:51:41.656127 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:41.970767 21091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:41.971104 21091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:41.971235 21091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:41.978434 21091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:51:41.978562 21091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:41.978643 21091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:41.978722 21091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:51:41.978782 21091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:51:41.988651 21091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:41.989939 21091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:41.991274 21091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:42.002440 21097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:42.002477 21099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:42.002699 21096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:42.003302 21091 server_base.cc:1034] running on GCE node
I20260929 20:51:42.004000 21091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:42.005327 21091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:42.006520 21091 hybrid_clock.cc:648] HybridClock initialized: now 1790715102006463 us; error 44 us; skew 500 ppm
I20260929 20:51:42.006850 21091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:42 dist-test-slave-blv7 krb5kdc[21078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:42.009752 21091 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:51:42.011173 21091 webserver.cc:460] Webserver started at http://127.0.0.1:40449/ using document root <none> and password file <none>
I20260929 20:51:42.011831 21091 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:42.011941 21091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:42.012225 21091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:42.015004 21091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "b65fb42ec1b34bc7af31ddb38e78bb92"
format_stamp: "Formatted at 2026-09-29 20:51:42 on dist-test-slave-blv7"
server_key: "de77e1f187bcfeab9aa90001a62d8cc3"
server_key_iv: "a1905b9ae4aae126ccd377efbca23951"
server_key_version: "encryptionkey@0"
I20260929 20:51:42.015780 21091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "b65fb42ec1b34bc7af31ddb38e78bb92"
format_stamp: "Formatted at 2026-09-29 20:51:42 on dist-test-slave-blv7"
server_key: "de77e1f187bcfeab9aa90001a62d8cc3"
server_key_iv: "a1905b9ae4aae126ccd377efbca23951"
server_key_version: "encryptionkey@0"
I20260929 20:51:42.020709 21091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:51:42.023947 21106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:42.025178 21091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:51:42.025332 21091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "b65fb42ec1b34bc7af31ddb38e78bb92"
format_stamp: "Formatted at 2026-09-29 20:51:42 on dist-test-slave-blv7"
server_key: "de77e1f187bcfeab9aa90001a62d8cc3"
server_key_iv: "a1905b9ae4aae126ccd377efbca23951"
server_key_version: "encryptionkey@0"
I20260929 20:51:42.025499 21091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:42.052201 21091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:42.056082 21091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:42.056334 21091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:42.073123 21091 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32781
I20260929 20:51:42.073158 21157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32781 every 8 connection(s)
I20260929 20:51:42.074425 21091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:51:42.077203 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 21091
I20260929 20:51:42.077356 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:51:42.077749 388 external_mini_cluster.cc:1428] Setting key f45dcbdbad96d481b0832a2b8c07a6e9
I20260929 20:51:42.078114 21158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:42.090013 21158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92: Bootstrap starting.
Sep 29 20:51:42 dist-test-slave-blv7 krb5kdc[21078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 20:51:42.093612 21158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:42.094777 21158 log.cc:824] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:42.097771 21158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92: No bootstrap required, opened a new log
I20260929 20:51:42.099900 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:42.079433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58141 (local address 127.0.0.1:32781)
0929 20:51:42.080292 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:42.080305 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:42.080349 (+ 44us) server_negotiation.cc:407] Connection header received
0929 20:51:42.082621 (+ 2272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:42.082645 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:42.083089 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:42.083482 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:42.084946 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:42.086181 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:42.087484 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:42.088184 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:42.088349 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:42.092709 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:42.092717 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:42.092731 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:42.092767 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:42.096019 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:42.096760 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:42.096766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:42.096772 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:42.096931 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:42.097287 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:42.097290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:42.097292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:42.097793 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:42.098056 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:42.098496 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 20:51:42.098842 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":407,"threads_started":1}
I20260929 20:51:42.103343 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:51:42.106714 21158 raft_consensus.cc:348] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65fb42ec1b34bc7af31ddb38e78bb92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32781 } }
I20260929 20:51:42.107225 21158 raft_consensus.cc:374] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:42.107366 21158 raft_consensus.cc:729] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b65fb42ec1b34bc7af31ddb38e78bb92, State: Initialized, Role: FOLLOWER
I20260929 20:51:42.108014 21158 consensus_queue.cc:260] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65fb42ec1b34bc7af31ddb38e78bb92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32781 } }
I20260929 20:51:42.108320 21158 raft_consensus.cc:388] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:51:42.108476 21158 raft_consensus.cc:482] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:51:42.108703 21158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:42.111294 21158 raft_consensus.cc:504] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65fb42ec1b34bc7af31ddb38e78bb92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32781 } }
I20260929 20:51:42.111853 21158 leader_election.cc:302] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b65fb42ec1b34bc7af31ddb38e78bb92; no voters:
I20260929 20:51:42.112390 21158 leader_election.cc:288] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:51:42.112524 21168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:42.114306 21158 sys_catalog.cc:564] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:51:42.114881 21168 raft_consensus.cc:686] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92 [term 1 LEADER]: Becoming Leader. State: Replica: b65fb42ec1b34bc7af31ddb38e78bb92, State: Running, Role: LEADER
I20260929 20:51:42.116828 21168 consensus_queue.cc:237] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65fb42ec1b34bc7af31ddb38e78bb92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32781 } }
I20260929 20:51:42.120078 21170 sys_catalog.cc:455] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b65fb42ec1b34bc7af31ddb38e78bb92. Latest consensus state: current_term: 1 leader_uuid: "b65fb42ec1b34bc7af31ddb38e78bb92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65fb42ec1b34bc7af31ddb38e78bb92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32781 } } }
I20260929 20:51:42.120460 21170 sys_catalog.cc:458] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:42.120986 21169 sys_catalog.cc:455] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b65fb42ec1b34bc7af31ddb38e78bb92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b65fb42ec1b34bc7af31ddb38e78bb92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32781 } } }
I20260929 20:51:42.121232 21169 sys_catalog.cc:458] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:42.124471 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:42.108418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35507 (local address 127.0.0.1:32781)
0929 20:51:42.108604 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:42.108610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:42.108930 (+ 320us) server_negotiation.cc:407] Connection header received
0929 20:51:42.109077 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:42.109082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:42.109161 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:42.109275 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:42.110516 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:42.111538 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:42.113059 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:42.113759 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:42.113860 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:42.116533 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:42.116540 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:42.116545 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:42.116584 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:42.120540 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:42.121470 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:42.121474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:42.121477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:42.121614 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:42.122080 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:42.122085 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:42.122089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:42.122513 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:42.123741 (+ 1228us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:42.123885 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 20:51:42.124115 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 20:51:42.132980 21185 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:51:42.133095 21185 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:51:42.133216 21178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:51:42.134814 21178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:51:42.142524 21178 catalog_manager.cc:1347] Generated new cluster ID: f572db5a60e848438608d7d2c84951ee
I20260929 20:51:42.142720 21178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:51:42.161302 21178 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:51:42.162564 21178 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:51:42.175519 21178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b65fb42ec1b34bc7af31ddb38e78bb92: Generated new TSK 0
I20260929 20:51:42.176333 21178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:51:42.183547 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 21091
I20260929 20:51:42.184598 21153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:42.275619 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 21091
2026-09-29T20:51:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1855 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:51:42 dist-test-slave-blv7 krb5kdc[21191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:51:42 dist-test-slave-blv7 krb5kdc[21191](info): set up 2 sockets
Sep 29 20:51:42 dist-test-slave-blv7 krb5kdc[21191](info): commencing operation
krb5kdc: starting...
W20260929 20:51:43.278884 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:51:43 dist-test-slave-blv7 krb5kdc[21191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:51:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:51:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:51:43.511999 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38569
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:43.822935 21204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:43.823292 21204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:43.823436 21204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:43.830562 21204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:51:43.830679 21204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:43.830756 21204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:43.830832 21204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:51:43.830890 21204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:51:43.840842 21204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:43.842126 21204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:43.843429 21204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:43.854358 21210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:43.854423 21212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:43.854389 21209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:43.855214 21204 server_base.cc:1034] running on GCE node
I20260929 20:51:43.855916 21204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:43.857257 21204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:43.858461 21204 hybrid_clock.cc:648] HybridClock initialized: now 1790715103858412 us; error 38 us; skew 500 ppm
I20260929 20:51:43.858783 21204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:43 dist-test-slave-blv7 krb5kdc[21191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:43.861789 21204 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:51:43.863157 21204 webserver.cc:460] Webserver started at http://127.0.0.1:43991/ using document root <none> and password file <none>
I20260929 20:51:43.863790 21204 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:43.863902 21204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:43.864181 21204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:43.867065 21204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "fc372f943c84467ca23203307ca4f28d"
format_stamp: "Formatted at 2026-09-29 20:51:43 on dist-test-slave-blv7"
server_key: "66267d1c0c7027a4233cd9381e99f808"
server_key_iv: "d6d9627386f4376c84fb74709bd4f9aa"
server_key_version: "encryptionkey@0"
I20260929 20:51:43.867769 21204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "fc372f943c84467ca23203307ca4f28d"
format_stamp: "Formatted at 2026-09-29 20:51:43 on dist-test-slave-blv7"
server_key: "66267d1c0c7027a4233cd9381e99f808"
server_key_iv: "d6d9627386f4376c84fb74709bd4f9aa"
server_key_version: "encryptionkey@0"
I20260929 20:51:43.872632 21204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:51:43.875914 21219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:43.877166 21204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:51:43.877321 21204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "fc372f943c84467ca23203307ca4f28d"
format_stamp: "Formatted at 2026-09-29 20:51:43 on dist-test-slave-blv7"
server_key: "66267d1c0c7027a4233cd9381e99f808"
server_key_iv: "d6d9627386f4376c84fb74709bd4f9aa"
server_key_version: "encryptionkey@0"
I20260929 20:51:43.877491 21204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:43.896849 21204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:43.901126 21204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:43.901384 21204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:43.918972 21204 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38569
I20260929 20:51:43.918983 21270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38569 every 8 connection(s)
I20260929 20:51:43.920392 21204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:51:43.922438 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 21204
I20260929 20:51:43.922608 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:51:43.923005 388 external_mini_cluster.cc:1428] Setting key 4c0c5736265a0d8e0916f31234b3d222
I20260929 20:51:43.924628 21271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:43.937516 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:43.924865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54983 (local address 127.0.0.1:38569)
0929 20:51:43.925652 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:43.925668 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:43.925717 (+ 49us) server_negotiation.cc:407] Connection header received
0929 20:51:43.927763 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:43.927793 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:43.928335 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:43.928807 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:43.929304 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:43.931302 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:43.932584 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:43.933565 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:43.933747 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:43.934704 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:43.934714 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:43.934736 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:43.935519 (+ 783us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:43.935611 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:43.935639 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:43.935755 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:43.935944 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 20:51:43.936222 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":303,"threads_started":1}
I20260929 20:51:43.938853 21271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d: Bootstrap starting.
I20260929 20:51:43.941061 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:51:43.942173 21271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:43.943374 21271 log.cc:824] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:43.946184 21271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d: No bootstrap required, opened a new log
I20260929 20:51:43.952318 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:43.945902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33235 (local address 127.0.0.1:38569)
0929 20:51:43.946070 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:43.946076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:43.946564 (+ 488us) server_negotiation.cc:407] Connection header received
0929 20:51:43.946706 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:43.946711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:43.946792 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:43.946906 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:43.947401 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:43.948492 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:43.949833 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:43.950499 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:43.950628 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:43.951626 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:43.951632 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:43.951636 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:43.951657 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:43.951702 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:43.951722 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:43.951813 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:43.951883 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 20:51:43.951983 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:51:43.955392 21271 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc372f943c84467ca23203307ca4f28d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } }
I20260929 20:51:43.955760 21271 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:43.955830 21271 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc372f943c84467ca23203307ca4f28d, State: Initialized, Role: FOLLOWER
I20260929 20:51:43.956341 21271 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc372f943c84467ca23203307ca4f28d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } }
I20260929 20:51:43.956534 21271 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:51:43.956624 21271 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:51:43.956741 21271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:43.958806 21271 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc372f943c84467ca23203307ca4f28d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } }
I20260929 20:51:43.959348 21271 leader_election.cc:302] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc372f943c84467ca23203307ca4f28d; no voters:
I20260929 20:51:43.959990 21271 leader_election.cc:288] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:51:43.960093 21281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:43.960745 21281 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d [term 1 LEADER]: Becoming Leader. State: Replica: fc372f943c84467ca23203307ca4f28d, State: Running, Role: LEADER
I20260929 20:51:43.961283 21281 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc372f943c84467ca23203307ca4f28d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } }
I20260929 20:51:43.961968 21271 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:51:43.964098 21282 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc372f943c84467ca23203307ca4f28d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc372f943c84467ca23203307ca4f28d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } } }
I20260929 20:51:43.964169 21283 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc372f943c84467ca23203307ca4f28d. Latest consensus state: current_term: 1 leader_uuid: "fc372f943c84467ca23203307ca4f28d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc372f943c84467ca23203307ca4f28d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38569 } } }
I20260929 20:51:43.964568 21283 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:43.964525 21282 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:43.966257 21290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:51:43.970263 21290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:51:43.978291 21290 catalog_manager.cc:1347] Generated new cluster ID: c874b0618c9e44408f70da9b31354c49
I20260929 20:51:43.978410 21290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:51:43.985101 21290 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:51:43.986848 21290 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:51:44.003417 21290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc372f943c84467ca23203307ca4f28d: Generated new TSK 0
I20260929 20:51:44.004462 21290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:51:44.013118 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 21204
W20260929 20:51:44.014060 21266 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54983
W20260929 20:51:44.014678 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 21204: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:51:44.014825 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 21204
2026-09-29T20:51:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1738 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:51:44 dist-test-slave-blv7 krb5kdc[21302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:51:44 dist-test-slave-blv7 krb5kdc[21302](info): set up 2 sockets
Sep 29 20:51:44 dist-test-slave-blv7 krb5kdc[21302](info): commencing operation
krb5kdc: starting...
W20260929 20:51:45.030997 388 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:51:45 dist-test-slave-blv7 krb5kdc[21302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:51:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:51:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:51:45.258426 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45937
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:45.572011 21315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:45.572304 21315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:45.572441 21315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:45.579505 21315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:51:45.579623 21315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:45.579697 21315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:45.579778 21315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:51:45.579835 21315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:51:45.589618 21315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:45.590895 21315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:45.592281 21315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:45.603511 21320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:45.603582 21323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:45.603605 21321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:45.604462 21315 server_base.cc:1034] running on GCE node
I20260929 20:51:45.605159 21315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:45.606442 21315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:45.607620 21315 hybrid_clock.cc:648] HybridClock initialized: now 1790715105607568 us; error 45 us; skew 500 ppm
I20260929 20:51:45.607942 21315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:45 dist-test-slave-blv7 krb5kdc[21302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:45.613813 21315 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:51:45.615180 21315 webserver.cc:460] Webserver started at http://127.0.0.1:37565/ using document root <none> and password file <none>
I20260929 20:51:45.615797 21315 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:45.615890 21315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:45.616120 21315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:45.618947 21315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "0cc5642743124c9792ab2b16a3420d94"
format_stamp: "Formatted at 2026-09-29 20:51:45 on dist-test-slave-blv7"
server_key: "b9e355457d9b6dec62df4e591975e910"
server_key_iv: "17f27412da9b8aacdd80516b3f607218"
server_key_version: "encryptionkey@0"
I20260929 20:51:45.619666 21315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "0cc5642743124c9792ab2b16a3420d94"
format_stamp: "Formatted at 2026-09-29 20:51:45 on dist-test-slave-blv7"
server_key: "b9e355457d9b6dec62df4e591975e910"
server_key_iv: "17f27412da9b8aacdd80516b3f607218"
server_key_version: "encryptionkey@0"
I20260929 20:51:45.624464 21315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:51:45.627717 21330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:45.628998 21315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:51:45.629149 21315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "0cc5642743124c9792ab2b16a3420d94"
format_stamp: "Formatted at 2026-09-29 20:51:45 on dist-test-slave-blv7"
server_key: "b9e355457d9b6dec62df4e591975e910"
server_key_iv: "17f27412da9b8aacdd80516b3f607218"
server_key_version: "encryptionkey@0"
I20260929 20:51:45.629319 21315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:45.634780 21315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:45.638854 21315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:45.639133 21315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:45.655846 21315 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45937
I20260929 20:51:45.655882 21381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45937 every 8 connection(s)
I20260929 20:51:45.657146 21315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:51:45.659044 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 21315
I20260929 20:51:45.659201 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:51:45.659652 388 external_mini_cluster.cc:1428] Setting key 93c97f6f57b147c648f56473335fc33a
I20260929 20:51:45.661077 21382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:45.667822 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:45.661378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53913 (local address 127.0.0.1:45937)
0929 20:51:45.662152 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:45.662168 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:45.662218 (+ 50us) server_negotiation.cc:407] Connection header received
0929 20:51:45.664094 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:45.664121 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:45.664571 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:45.664993 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:45.665943 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:45.665952 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:45.665974 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:45.665990 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:45.666079 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:45.666109 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:45.666312 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:45.666433 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 20:51:45.666622 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":318,"threads_started":1}
I20260929 20:51:45.671684 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:51:45.675238 21382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94: Bootstrap starting.
I20260929 20:51:45.679082 21382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:45.679178 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:45.676778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43303 (local address 127.0.0.1:45937)
0929 20:51:45.676975 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:45.676982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:45.677240 (+ 258us) server_negotiation.cc:407] Connection header received
0929 20:51:45.677373 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:45.677378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:45.677446 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:45.677584 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:45.678461 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:45.678467 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:45.678471 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:45.678489 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:45.678547 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:45.678572 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:45.678697 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:45.678802 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 20:51:45.678861 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:51:45.680385 21382 log.cc:824] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:45.683154 21382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94: No bootstrap required, opened a new log
I20260929 20:51:45.690325 21382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5642743124c9792ab2b16a3420d94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45937 } }
I20260929 20:51:45.690694 21382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:45.690796 21382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cc5642743124c9792ab2b16a3420d94, State: Initialized, Role: FOLLOWER
I20260929 20:51:45.691434 21382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5642743124c9792ab2b16a3420d94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45937 } }
I20260929 20:51:45.691697 21382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:51:45.691839 21382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:51:45.692009 21382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:45.694115 21382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5642743124c9792ab2b16a3420d94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45937 } }
I20260929 20:51:45.694604 21382 leader_election.cc:302] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cc5642743124c9792ab2b16a3420d94; no voters:
I20260929 20:51:45.695220 21382 leader_election.cc:288] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:51:45.695348 21392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:45.695923 21392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94 [term 1 LEADER]: Becoming Leader. State: Replica: 0cc5642743124c9792ab2b16a3420d94, State: Running, Role: LEADER
I20260929 20:51:45.696447 21392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5642743124c9792ab2b16a3420d94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45937 } }
I20260929 20:51:45.697173 21382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:51:45.698812 21394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cc5642743124c9792ab2b16a3420d94. Latest consensus state: current_term: 1 leader_uuid: "0cc5642743124c9792ab2b16a3420d94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5642743124c9792ab2b16a3420d94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45937 } } }
I20260929 20:51:45.699297 21394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:45.699261 21393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cc5642743124c9792ab2b16a3420d94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc5642743124c9792ab2b16a3420d94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45937 } } }
I20260929 20:51:45.699535 21393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:45.701471 21397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:51:45.706573 21397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:51:45.714885 21397 catalog_manager.cc:1347] Generated new cluster ID: 79d111eabe8f4563828d61389f39e9c0
I20260929 20:51:45.714988 21397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:51:45.730551 21397 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:51:45.730939 21397 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:51:45.739671 21397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cc5642743124c9792ab2b16a3420d94: Generated new TSK 0
I20260929 20:51:45.740408 21397 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:51:45.753741 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 21315
W20260929 20:51:45.754766 21377 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53913
W20260929 20:51:45.755378 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 21315: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:51:45.755532 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 21315
2026-09-29T20:51:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1743 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:51:45 dist-test-slave-blv7 krb5kdc[21413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:51:45 dist-test-slave-blv7 krb5kdc[21413](info): set up 2 sockets
Sep 29 20:51:45 dist-test-slave-blv7 krb5kdc[21413](info): commencing operation
krb5kdc: starting...
W20260929 20:51:46.759228 388 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:51:46 dist-test-slave-blv7 krb5kdc[21413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:51:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:51:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:51:46.990383 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41171
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:47.298893 21426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:47.299192 21426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:47.299320 21426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:47.306495 21426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:51:47.306602 21426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:47.306666 21426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:47.306749 21426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:51:47.306805 21426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:51:47.316704 21426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:47.317916 21426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:47.319188 21426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:47.329849 21431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:47.329901 21432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:47.329873 21434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:47.330658 21426 server_base.cc:1034] running on GCE node
I20260929 20:51:47.331349 21426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:47.332590 21426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:47.333777 21426 hybrid_clock.cc:648] HybridClock initialized: now 1790715107333733 us; error 40 us; skew 500 ppm
I20260929 20:51:47.334075 21426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:47 dist-test-slave-blv7 krb5kdc[21413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:47.336862 21426 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:51:47.338205 21426 webserver.cc:460] Webserver started at http://127.0.0.1:34327/ using document root <none> and password file <none>
I20260929 20:51:47.338812 21426 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:47.338907 21426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:47.339188 21426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:47.341989 21426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "b7b97ef7906347aa8a76063fa09732cf"
format_stamp: "Formatted at 2026-09-29 20:51:47 on dist-test-slave-blv7"
server_key: "67673bea9d91faeb20efd8d44a6767f4"
server_key_iv: "e3a45c4297fded9c471cad131e7c63ed"
server_key_version: "encryptionkey@0"
I20260929 20:51:47.342675 21426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "b7b97ef7906347aa8a76063fa09732cf"
format_stamp: "Formatted at 2026-09-29 20:51:47 on dist-test-slave-blv7"
server_key: "67673bea9d91faeb20efd8d44a6767f4"
server_key_iv: "e3a45c4297fded9c471cad131e7c63ed"
server_key_version: "encryptionkey@0"
I20260929 20:51:47.347447 21426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260929 20:51:47.350600 21441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:47.351832 21426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:51:47.351969 21426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "b7b97ef7906347aa8a76063fa09732cf"
format_stamp: "Formatted at 2026-09-29 20:51:47 on dist-test-slave-blv7"
server_key: "67673bea9d91faeb20efd8d44a6767f4"
server_key_iv: "e3a45c4297fded9c471cad131e7c63ed"
server_key_version: "encryptionkey@0"
I20260929 20:51:47.352116 21426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:47.368376 21426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:47.372334 21426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:47.372601 21426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:47.387902 21426 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41171
I20260929 20:51:47.387923 21492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41171 every 8 connection(s)
I20260929 20:51:47.389200 21426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:51:47.390683 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 21426
I20260929 20:51:47.390834 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:51:47.391283 388 external_mini_cluster.cc:1428] Setting key 4d4d11c0b7bbd0c10ac5f2fe604d4dde
I20260929 20:51:47.393183 21493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:51:47 dist-test-slave-blv7 krb5kdc[21413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 20:51:47.405504 21493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf: Bootstrap starting.
I20260929 20:51:47.408743 21493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:47.409806 21493 log.cc:824] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:47.412881 21493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf: No bootstrap required, opened a new log
I20260929 20:51:47.413631 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:47.392908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:47089 (local address 127.0.0.1:41171)
0929 20:51:47.393646 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:47.393659 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:47.393707 (+ 48us) server_negotiation.cc:407] Connection header received
0929 20:51:47.395804 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:47.395830 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:47.396325 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:47.396762 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:47.397975 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:47.399946 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:47.401174 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:47.402169 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:47.402360 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:47.406433 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:47.406442 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:47.406458 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:47.406507 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:47.409447 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:47.410401 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:47.410410 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:47.410419 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:47.410664 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:47.411106 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:47.411111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:47.411115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:47.411609 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:47.411832 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:47.412322 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 20:51:47.412607 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":280,"threads_started":1}
I20260929 20:51:47.417337 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:51:47.421005 21493 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b97ef7906347aa8a76063fa09732cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } }
I20260929 20:51:47.421541 21493 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:47.421653 21493 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b97ef7906347aa8a76063fa09732cf, State: Initialized, Role: FOLLOWER
I20260929 20:51:47.422206 21493 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b97ef7906347aa8a76063fa09732cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } }
I20260929 20:51:47.422420 21493 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:51:47.422533 21493 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:51:47.422675 21493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:47.424772 21493 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b97ef7906347aa8a76063fa09732cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } }
I20260929 20:51:47.425223 21493 leader_election.cc:302] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7b97ef7906347aa8a76063fa09732cf; no voters:
I20260929 20:51:47.425788 21493 leader_election.cc:288] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:51:47.425923 21503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:47.428028 21493 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:51:47.430971 21503 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf [term 1 LEADER]: Becoming Leader. State: Replica: b7b97ef7906347aa8a76063fa09732cf, State: Running, Role: LEADER
I20260929 20:51:47.431700 21503 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b97ef7906347aa8a76063fa09732cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } }
I20260929 20:51:47.434592 21505 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7b97ef7906347aa8a76063fa09732cf. Latest consensus state: current_term: 1 leader_uuid: "b7b97ef7906347aa8a76063fa09732cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b97ef7906347aa8a76063fa09732cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } } }
I20260929 20:51:47.434950 21505 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:47.435954 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:47.421353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:41199 (local address 127.0.0.1:41171)
0929 20:51:47.421485 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:47.421489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:47.421942 (+ 453us) server_negotiation.cc:407] Connection header received
0929 20:51:47.422115 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:47.422119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:47.422175 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:47.422254 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:47.423683 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:47.424424 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:47.425690 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:47.426294 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:47.427044 (+ 750us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:47.428877 (+ 1833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:47.428882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:47.428885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:47.428912 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:47.432291 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:47.433009 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:47.433014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:47.433016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:47.433132 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:47.433630 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:47.433635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:47.433637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:47.434006 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:47.435193 (+ 1187us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:47.435364 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 20:51:47.435570 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 20:51:47.436007 21504 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7b97ef7906347aa8a76063fa09732cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7b97ef7906347aa8a76063fa09732cf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } } }
I20260929 20:51:47.436779 21504 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf [sys.catalog]: This master's current role is: LEADER
W20260929 20:51:47.443881 21520 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:51:47.444010 21520 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:51:47.444120 21514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:51:47.446317 21514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:51:47.455474 21514 catalog_manager.cc:1347] Generated new cluster ID: 2cd25d5282164d67806bbe80818740a9
I20260929 20:51:47.455590 21514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:51:47.481904 21514 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:51:47.483743 21514 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:51:47.491647 21514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7b97ef7906347aa8a76063fa09732cf: Generated new TSK 0
I20260929 20:51:47.492472 21514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:51:47.504366 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 21426
I20260929 20:51:47.506260 21488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:47.598364 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 21426
W20260929 20:51:47.612104 21411 connection.cc:535] client connection to 127.0.0.1:41171 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41171): Connection reset by peer (error 104)
2026-09-29T20:51:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1840 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:51:47 dist-test-slave-blv7 krb5kdc[21526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:51:47 dist-test-slave-blv7 krb5kdc[21526](info): set up 2 sockets
Sep 29 20:51:47 dist-test-slave-blv7 krb5kdc[21526](info): commencing operation
krb5kdc: starting...
W20260929 20:51:48.611094 388 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:51:48 dist-test-slave-blv7 krb5kdc[21526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:51:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:51:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:51:48.840047 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43861
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:49.152349 21539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:49.152631 21539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:49.152721 21539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:49.159736 21539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:51:49.159840 21539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:49.159891 21539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:49.159961 21539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:51:49.160023 21539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:51:49.169765 21539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:49.171068 21539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:49.172322 21539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:49.183645 21547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:49.183681 21545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:49.183904 21544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:49.184527 21539 server_base.cc:1034] running on GCE node
I20260929 20:51:49.185241 21539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:49.186581 21539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:49.187767 21539 hybrid_clock.cc:648] HybridClock initialized: now 1790715109187718 us; error 41 us; skew 500 ppm
I20260929 20:51:49.188071 21539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:49 dist-test-slave-blv7 krb5kdc[21526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:49.190968 21539 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:51:49.192348 21539 webserver.cc:460] Webserver started at http://127.0.0.1:36985/ using document root <none> and password file <none>
I20260929 20:51:49.192993 21539 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:49.193099 21539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:49.193377 21539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:49.196280 21539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "a3aec511acdd4e528a7daaea89e4de16"
format_stamp: "Formatted at 2026-09-29 20:51:49 on dist-test-slave-blv7"
server_key: "f9aa7268ee5b0585725d72989a09adc3"
server_key_iv: "a5172e7f34674e661947d56647e75b60"
server_key_version: "encryptionkey@0"
I20260929 20:51:49.196996 21539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "a3aec511acdd4e528a7daaea89e4de16"
format_stamp: "Formatted at 2026-09-29 20:51:49 on dist-test-slave-blv7"
server_key: "f9aa7268ee5b0585725d72989a09adc3"
server_key_iv: "a5172e7f34674e661947d56647e75b60"
server_key_version: "encryptionkey@0"
I20260929 20:51:49.201792 21539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 20:51:49.205049 21554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:49.206286 21539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:51:49.206430 21539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "a3aec511acdd4e528a7daaea89e4de16"
format_stamp: "Formatted at 2026-09-29 20:51:49 on dist-test-slave-blv7"
server_key: "f9aa7268ee5b0585725d72989a09adc3"
server_key_iv: "a5172e7f34674e661947d56647e75b60"
server_key_version: "encryptionkey@0"
I20260929 20:51:49.206598 21539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:49.230346 21539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:49.234159 21539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:49.234401 21539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:49.250201 21539 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43861
I20260929 20:51:49.250221 21605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43861 every 8 connection(s)
I20260929 20:51:49.251495 21539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:51:49.254985 21606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:49.261797 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 21539
I20260929 20:51:49.261986 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:51:49.262461 388 external_mini_cluster.cc:1428] Setting key d3805842c4712faf587758b2b02387e9
I20260929 20:51:49.265434 21606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16: Bootstrap starting.
I20260929 20:51:49.268584 21606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:49.269632 21606 log.cc:824] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:49.272491 21606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16: No bootstrap required, opened a new log
I20260929 20:51:49.274950 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:49.264154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:50227 (local address 127.0.0.1:43861)
0929 20:51:49.264806 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:49.264817 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:49.264856 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:51:49.266510 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:49.266531 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:49.266923 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:49.267288 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:49.267789 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:49.269082 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:49.270818 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:49.271493 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:49.271613 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:49.272803 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:49.272811 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:49.272824 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:49.273343 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:49.273415 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:49.273444 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:49.273561 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:49.273813 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 20:51:49.274004 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":289,"threads_started":1}
I20260929 20:51:49.278654 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:51:49.281489 21606 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3aec511acdd4e528a7daaea89e4de16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43861 } }
I20260929 20:51:49.281931 21606 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:49.282038 21606 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3aec511acdd4e528a7daaea89e4de16, State: Initialized, Role: FOLLOWER
I20260929 20:51:49.282678 21606 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3aec511acdd4e528a7daaea89e4de16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43861 } }
I20260929 20:51:49.282949 21606 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:51:49.283105 21606 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:51:49.283274 21606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:49.285907 21606 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3aec511acdd4e528a7daaea89e4de16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43861 } }
I20260929 20:51:49.286437 21606 leader_election.cc:302] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3aec511acdd4e528a7daaea89e4de16; no voters:
I20260929 20:51:49.287084 21606 leader_election.cc:288] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:51:49.287145 21616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:49.289149 21606 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:51:49.290673 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:49.283617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:45905 (local address 127.0.0.1:43861)
0929 20:51:49.283789 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:49.283795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:49.284256 (+ 461us) server_negotiation.cc:407] Connection header received
0929 20:51:49.284410 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:49.284414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:49.284474 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:49.284549 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:49.285091 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:49.285770 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:49.287549 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:49.288211 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:49.288313 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:49.289778 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:49.289786 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:49.289791 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:49.289814 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:49.289893 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:49.289915 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:49.290037 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:49.290260 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 20:51:49.290314 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:51:49.293687 21616 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16 [term 1 LEADER]: Becoming Leader. State: Replica: a3aec511acdd4e528a7daaea89e4de16, State: Running, Role: LEADER
I20260929 20:51:49.294124 21616 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3aec511acdd4e528a7daaea89e4de16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43861 } }
I20260929 20:51:49.296573 21617 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3aec511acdd4e528a7daaea89e4de16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3aec511acdd4e528a7daaea89e4de16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43861 } } }
I20260929 20:51:49.297055 21617 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:49.297289 21618 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3aec511acdd4e528a7daaea89e4de16. Latest consensus state: current_term: 1 leader_uuid: "a3aec511acdd4e528a7daaea89e4de16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3aec511acdd4e528a7daaea89e4de16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43861 } } }
I20260929 20:51:49.297617 21618 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:49.302719 21625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:51:49.306604 21625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:51:49.314532 21625 catalog_manager.cc:1347] Generated new cluster ID: b2e3338253d443a9b9e2f9e39c960000
I20260929 20:51:49.314631 21625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:51:49.326241 21625 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:51:49.327811 21625 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:51:49.341796 21625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3aec511acdd4e528a7daaea89e4de16: Generated new TSK 0
I20260929 20:51:49.342854 21625 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:51:49.350934 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 21539
W20260929 20:51:49.352021 21601 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.17:50227
W20260929 20:51:49.352749 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 21539: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:51:49.352892 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 21539
W20260929 20:51:49.366046 21524 connection.cc:535] client connection to 127.0.0.1:43861 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43861): Connection reset by peer (error 104)
2026-09-29T20:51:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1753 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:51:49 dist-test-slave-blv7 krb5kdc[21637](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:51:49 dist-test-slave-blv7 krb5kdc[21637](info): set up 2 sockets
Sep 29 20:51:49 dist-test-slave-blv7 krb5kdc[21637](info): commencing operation
krb5kdc: starting...
W20260929 20:51:50.355082 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:51:50 dist-test-slave-blv7 krb5kdc[21637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:51:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:51:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:51:50.584429 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43101
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:50.892864 21650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:50.893136 21650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:50.893260 21650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:50.900257 21650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:51:50.900352 21650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:50.900418 21650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:50.900512 21650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:51:50.900571 21650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:51:50.909960 21650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:50.911226 21650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:50.912427 21650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:50.923130 21656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:50.923480 21658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:50.923570 21655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:50.923926 21650 server_base.cc:1034] running on GCE node
I20260929 20:51:50.924621 21650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:50.925926 21650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:50.927109 21650 hybrid_clock.cc:648] HybridClock initialized: now 1790715110927053 us; error 45 us; skew 500 ppm
I20260929 20:51:50.927416 21650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:50 dist-test-slave-blv7 krb5kdc[21637](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:50.935896 21650 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:51:50.937263 21650 webserver.cc:460] Webserver started at http://127.0.0.1:43309/ using document root <none> and password file <none>
I20260929 20:51:50.937922 21650 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:50.938036 21650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:50.938328 21650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:50.941246 21650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "6325198f6ba84e32a910e10a678a5659"
format_stamp: "Formatted at 2026-09-29 20:51:50 on dist-test-slave-blv7"
server_key: "481bc8a53e4abd8c9f5de04b0ff24379"
server_key_iv: "8821be2e7ea8e7d323dda79384b9327e"
server_key_version: "encryptionkey@0"
I20260929 20:51:50.941967 21650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "6325198f6ba84e32a910e10a678a5659"
format_stamp: "Formatted at 2026-09-29 20:51:50 on dist-test-slave-blv7"
server_key: "481bc8a53e4abd8c9f5de04b0ff24379"
server_key_iv: "8821be2e7ea8e7d323dda79384b9327e"
server_key_version: "encryptionkey@0"
I20260929 20:51:50.947089 21650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 20:51:50.950307 21665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:50.951574 21650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 20:51:50.951730 21650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "6325198f6ba84e32a910e10a678a5659"
format_stamp: "Formatted at 2026-09-29 20:51:50 on dist-test-slave-blv7"
server_key: "481bc8a53e4abd8c9f5de04b0ff24379"
server_key_iv: "8821be2e7ea8e7d323dda79384b9327e"
server_key_version: "encryptionkey@0"
I20260929 20:51:50.951902 21650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:50.957195 21650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:50.960750 21650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:50.961011 21650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:50.977429 21650 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43101
I20260929 20:51:50.977459 21716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43101 every 8 connection(s)
I20260929 20:51:50.978750 21650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:51:50.982395 21717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:50.984419 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 21650
I20260929 20:51:50.984571 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:51:50.984968 388 external_mini_cluster.cc:1428] Setting key 6231e28f146097a6b577ca6125d86953
I20260929 20:51:50.992720 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:50.986782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:47929 (local address 127.0.0.1:43101)
0929 20:51:50.987861 (+ 1079us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:50.987872 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:50.987912 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:51:50.989505 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:50.989526 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:50.989872 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:50.990196 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:50.991213 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:50.991219 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:50.991232 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:50.991253 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:50.991323 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:50.991345 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:50.991514 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:50.991612 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 20:51:50.991782 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":612,"threads_started":1}
I20260929 20:51:50.993429 21717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659: Bootstrap starting.
I20260929 20:51:50.996234 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:51:50.996665 21717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:50.997587 21717 log.cc:824] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:51.000692 21717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659: No bootstrap required, opened a new log
I20260929 20:51:51.002717 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:51.000406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:50171 (local address 127.0.0.1:43101)
0929 20:51:51.000576 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:51.000581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:51.000993 (+ 412us) server_negotiation.cc:407] Connection header received
0929 20:51:51.001166 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:51.001170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:51.001218 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:51.001305 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:51.002113 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:51.002119 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:51.002124 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:51.002141 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:51.002180 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:51.002198 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:51.002320 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:51.002393 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 20:51:51.002458 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 20:51:51.008232 21717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6325198f6ba84e32a910e10a678a5659" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43101 } }
I20260929 20:51:51.008581 21717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:51.008652 21717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6325198f6ba84e32a910e10a678a5659, State: Initialized, Role: FOLLOWER
I20260929 20:51:51.009177 21717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6325198f6ba84e32a910e10a678a5659" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43101 } }
I20260929 20:51:51.009374 21717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:51:51.009523 21717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:51:51.009704 21717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:51.012120 21717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6325198f6ba84e32a910e10a678a5659" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43101 } }
I20260929 20:51:51.012574 21717 leader_election.cc:302] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6325198f6ba84e32a910e10a678a5659; no voters:
I20260929 20:51:51.013099 21717 leader_election.cc:288] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:51:51.013212 21727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:51.013780 21727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659 [term 1 LEADER]: Becoming Leader. State: Replica: 6325198f6ba84e32a910e10a678a5659, State: Running, Role: LEADER
I20260929 20:51:51.014315 21727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6325198f6ba84e32a910e10a678a5659" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43101 } }
I20260929 20:51:51.015007 21717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:51:51.017069 21728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6325198f6ba84e32a910e10a678a5659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6325198f6ba84e32a910e10a678a5659" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43101 } } }
I20260929 20:51:51.017493 21728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:51.017915 21729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6325198f6ba84e32a910e10a678a5659. Latest consensus state: current_term: 1 leader_uuid: "6325198f6ba84e32a910e10a678a5659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6325198f6ba84e32a910e10a678a5659" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43101 } } }
I20260929 20:51:51.018188 21729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:51.018396 21734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:51:51.023057 21734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:51:51.031862 21734 catalog_manager.cc:1347] Generated new cluster ID: b29e99145b714ac0872b65b8c3961e1c
I20260929 20:51:51.031965 21734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:51:51.045939 21734 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:51:51.046439 21734 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:51:51.063791 21734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6325198f6ba84e32a910e10a678a5659: Generated new TSK 0
I20260929 20:51:51.064910 21734 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:51:51.077060 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 21650
W20260929 20:51:51.078147 21712 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.17:47929
W20260929 20:51:51.078847 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 21650: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:51:51.078974 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 21650
2026-09-29T20:51:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1726 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:51:51 dist-test-slave-blv7 krb5kdc[21748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:51:51 dist-test-slave-blv7 krb5kdc[21748](info): set up 2 sockets
Sep 29 20:51:51 dist-test-slave-blv7 krb5kdc[21748](info): commencing operation
krb5kdc: starting...
W20260929 20:51:52.083180 388 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:51:52 dist-test-slave-blv7 krb5kdc[21748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:51:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:51:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:51:52.309860 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45925
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:52.623412 21761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:52.623742 21761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:52.623896 21761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:52.631410 21761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:51:52.631546 21761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:52.631634 21761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:52.631738 21761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:51:52.631803 21761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:51:52.642238 21761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45925
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:52.643623 21761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:52.644991 21761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:52.656728 21767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:52.656786 21766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:52.656816 21769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:52.657773 21761 server_base.cc:1034] running on GCE node
I20260929 20:51:52.658516 21761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:52.659974 21761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:52.661159 21761 hybrid_clock.cc:648] HybridClock initialized: now 1790715112661107 us; error 50 us; skew 500 ppm
I20260929 20:51:52.661489 21761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:52 dist-test-slave-blv7 krb5kdc[21748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:52.664496 21761 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:51:52.665872 21761 webserver.cc:460] Webserver started at http://127.0.0.1:43103/ using document root <none> and password file <none>
I20260929 20:51:52.666497 21761 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:52.666592 21761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:52.666832 21761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:52.669750 21761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "7af7772a23bb4e1c96a19e982ae8c5ef"
format_stamp: "Formatted at 2026-09-29 20:51:52 on dist-test-slave-blv7"
server_key: "314eb4b2348803c84c3ee469945e481a"
server_key_iv: "b472346578a33e2fd6a19c6c5b00cb67"
server_key_version: "encryptionkey@0"
I20260929 20:51:52.670440 21761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "7af7772a23bb4e1c96a19e982ae8c5ef"
format_stamp: "Formatted at 2026-09-29 20:51:52 on dist-test-slave-blv7"
server_key: "314eb4b2348803c84c3ee469945e481a"
server_key_iv: "b472346578a33e2fd6a19c6c5b00cb67"
server_key_version: "encryptionkey@0"
I20260929 20:51:52.675428 21761 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 20:51:52.678789 21776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:52.680158 21761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:51:52.680302 21761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "7af7772a23bb4e1c96a19e982ae8c5ef"
format_stamp: "Formatted at 2026-09-29 20:51:52 on dist-test-slave-blv7"
server_key: "314eb4b2348803c84c3ee469945e481a"
server_key_iv: "b472346578a33e2fd6a19c6c5b00cb67"
server_key_version: "encryptionkey@0"
I20260929 20:51:52.680488 21761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:52.696718 21761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:52.700671 21761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:52.700918 21761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:52.716881 21761 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45925
I20260929 20:51:52.716899 21827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45925 every 8 connection(s)
I20260929 20:51:52.718154 21761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:51:52.720738 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 21761
I20260929 20:51:52.720892 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:51:52.721284 388 external_mini_cluster.cc:1428] Setting key 1b649e981ea229e26614ce43be746230
I20260929 20:51:52.721768 21828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:52.733271 21828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef: Bootstrap starting.
Sep 29 20:51:52 dist-test-slave-blv7 krb5kdc[21748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 20:51:52.736223 21828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:52.737188 21828 log.cc:824] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:52.739728 21828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef: No bootstrap required, opened a new log
I20260929 20:51:52.741468 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:52.722946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:45203 (local address 127.0.0.1:45925)
0929 20:51:52.723680 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:52.723691 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:52.723729 (+ 38us) server_negotiation.cc:407] Connection header received
0929 20:51:52.725232 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:52.725252 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:52.725660 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:52.726014 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:52.727466 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:52.728744 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:52.729957 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:52.730643 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:52.730822 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:52.734689 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:52.734696 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:52.734710 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:52.734748 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:52.737631 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:52.738337 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:52.738344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:52.738350 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:52.738513 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:52.738953 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:52.738956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:52.738959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:52.739478 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:52.739791 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:52.740230 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 20:51:52.740534 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":287,"threads_started":1}
I20260929 20:51:52.744959 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:51:52.747206 21828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af7772a23bb4e1c96a19e982ae8c5ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45925 } }
I20260929 20:51:52.747648 21828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:52.747733 21828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af7772a23bb4e1c96a19e982ae8c5ef, State: Initialized, Role: FOLLOWER
I20260929 20:51:52.748289 21828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af7772a23bb4e1c96a19e982ae8c5ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45925 } }
I20260929 20:51:52.748509 21828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:51:52.748622 21828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:51:52.748778 21828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:52.751721 21828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af7772a23bb4e1c96a19e982ae8c5ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45925 } }
I20260929 20:51:52.752182 21828 leader_election.cc:302] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7af7772a23bb4e1c96a19e982ae8c5ef; no voters:
I20260929 20:51:52.752779 21828 leader_election.cc:288] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:51:52.752851 21838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:52.754791 21828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:51:52.755290 21838 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef [term 1 LEADER]: Becoming Leader. State: Replica: 7af7772a23bb4e1c96a19e982ae8c5ef, State: Running, Role: LEADER
I20260929 20:51:52.755836 21838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af7772a23bb4e1c96a19e982ae8c5ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45925 } }
I20260929 20:51:52.758729 21839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7af7772a23bb4e1c96a19e982ae8c5ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af7772a23bb4e1c96a19e982ae8c5ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45925 } } }
I20260929 20:51:52.759117 21839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:52.759589 21840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7af7772a23bb4e1c96a19e982ae8c5ef. Latest consensus state: current_term: 1 leader_uuid: "7af7772a23bb4e1c96a19e982ae8c5ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af7772a23bb4e1c96a19e982ae8c5ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45925 } } }
I20260929 20:51:52.759835 21840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:52.763648 21848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:51:52.765105 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:52.748805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:43675 (local address 127.0.0.1:45925)
0929 20:51:52.748977 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:52.748982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:52.749470 (+ 488us) server_negotiation.cc:407] Connection header received
0929 20:51:52.749624 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:52.749628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:52.749695 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:52.749772 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:52.751413 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:52.752193 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:52.753887 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:52.754520 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:52.754609 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:52.757804 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:52.757812 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:52.757820 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:52.757861 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:52.761668 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:52.762571 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:52.762577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:52.762581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:52.762774 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:52.763310 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:52.763316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:52.763319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:52.763729 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:52.764085 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:52.764543 (+ 458us) server_negotiation.cc:299] Negotiation successful
0929 20:51:52.764745 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:51:52.769039 21848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:51:52.777949 21848 catalog_manager.cc:1347] Generated new cluster ID: f1d0385f1e524dc5a81a553752030fff
I20260929 20:51:52.778040 21848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:51:52.800025 21848 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:51:52.801328 21848 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:51:52.811606 21848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7af7772a23bb4e1c96a19e982ae8c5ef: Generated new TSK 0
I20260929 20:51:52.812359 21848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:51:52.820631 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 21761
I20260929 20:51:52.822152 21823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:52.910132 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 21761
W20260929 20:51:52.923589 21746 connection.cc:535] client connection to 127.0.0.1:45925 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45925): Connection reset by peer (error 104)
2026-09-29T20:51:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1831 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:51:52 dist-test-slave-blv7 krb5kdc[21860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:51:52 dist-test-slave-blv7 krb5kdc[21860](info): set up 2 sockets
Sep 29 20:51:52 dist-test-slave-blv7 krb5kdc[21860](info): commencing operation
krb5kdc: starting...
W20260929 20:51:53.923275 388 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:51:54 dist-test-slave-blv7 krb5kdc[21860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:51:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:51:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:51:54.157948 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33573
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:54.471621 21873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:54.472059 21873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:54.472215 21873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:54.482667 21873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:51:54.482810 21873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:54.482890 21873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:54.482980 21873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:51:54.483064 21873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:51:54.496701 21873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33573
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:54.498545 21873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:54.500432 21873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:54.514933 21878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:54.514899 21879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:54.515225 21881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:54.515976 21873 server_base.cc:1034] running on GCE node
I20260929 20:51:54.516767 21873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:54.518155 21873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:54.519344 21873 hybrid_clock.cc:648] HybridClock initialized: now 1790715114519290 us; error 51 us; skew 500 ppm
I20260929 20:51:54.519673 21873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:54 dist-test-slave-blv7 krb5kdc[21860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:54.522598 21873 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:51:54.524183 21873 webserver.cc:460] Webserver started at http://127.0.0.1:36327/ using document root <none> and password file <none>
I20260929 20:51:54.524835 21873 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:54.524955 21873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:54.525228 21873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:54.528163 21873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "d4895b38e54f46428745ef398367cdc4"
format_stamp: "Formatted at 2026-09-29 20:51:54 on dist-test-slave-blv7"
server_key: "e0f3b774bccd0e090db7f2d0053e446b"
server_key_iv: "75316d71f5f2b570c261347bcecd2989"
server_key_version: "encryptionkey@0"
I20260929 20:51:54.528867 21873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "d4895b38e54f46428745ef398367cdc4"
format_stamp: "Formatted at 2026-09-29 20:51:54 on dist-test-slave-blv7"
server_key: "e0f3b774bccd0e090db7f2d0053e446b"
server_key_iv: "75316d71f5f2b570c261347bcecd2989"
server_key_version: "encryptionkey@0"
I20260929 20:51:54.533826 21873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:51:54.537052 21888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:54.538314 21873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:51:54.538460 21873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "d4895b38e54f46428745ef398367cdc4"
format_stamp: "Formatted at 2026-09-29 20:51:54 on dist-test-slave-blv7"
server_key: "e0f3b774bccd0e090db7f2d0053e446b"
server_key_iv: "75316d71f5f2b570c261347bcecd2989"
server_key_version: "encryptionkey@0"
I20260929 20:51:54.538630 21873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:54.561860 21873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:54.566183 21873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:54.566448 21873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:54.583530 21873 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33573
I20260929 20:51:54.583567 21939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33573 every 8 connection(s)
I20260929 20:51:54.584856 21873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:51:54.588708 21940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:54.589170 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 21873
I20260929 20:51:54.589332 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:51:54.589831 388 external_mini_cluster.cc:1428] Setting key cad99d5e96e72423279dd8fa2f146e41
I20260929 20:51:54.601058 21940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4: Bootstrap starting.
I20260929 20:51:54.603099 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:54.591737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:54793 (local address 127.0.0.1:33573)
0929 20:51:54.592487 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:54.592498 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:54.592539 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:51:54.594856 (+ 2317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:54.594882 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:54.595351 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:54.595723 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:54.596285 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:54.597544 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:54.598813 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:54.599536 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:54.599702 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:54.600906 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:54.600913 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:54.600930 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:54.601464 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:54.601544 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:54.601568 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:54.601694 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:54.601971 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 20:51:54.602122 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":320,"threads_started":1}
I20260929 20:51:54.604221 21940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:54.605302 21940 log.cc:824] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:54.606671 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:51:54.608062 21940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4: No bootstrap required, opened a new log
I20260929 20:51:54.615393 21940 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4895b38e54f46428745ef398367cdc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33573 } }
I20260929 20:51:54.615767 21940 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:54.615868 21940 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4895b38e54f46428745ef398367cdc4, State: Initialized, Role: FOLLOWER
I20260929 20:51:54.616391 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:54.610737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:57033 (local address 127.0.0.1:33573)
0929 20:51:54.610910 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:54.610916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:54.611184 (+ 268us) server_negotiation.cc:407] Connection header received
0929 20:51:54.611366 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:54.611369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:54.611438 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:54.611515 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:54.611979 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:54.612741 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:54.614021 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:54.614482 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:54.614592 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:54.615663 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:54.615667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:54.615671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:54.615686 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:54.615721 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:54.615737 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:54.615843 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:54.616054 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 20:51:54.616098 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 20:51:54.616529 21940 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4895b38e54f46428745ef398367cdc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33573 } }
I20260929 20:51:54.616824 21940 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:51:54.616969 21940 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:51:54.617121 21940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:54.619354 21940 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4895b38e54f46428745ef398367cdc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33573 } }
I20260929 20:51:54.619807 21940 leader_election.cc:302] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4895b38e54f46428745ef398367cdc4; no voters:
I20260929 20:51:54.620309 21940 leader_election.cc:288] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:51:54.620412 21950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:54.621050 21950 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4 [term 1 LEADER]: Becoming Leader. State: Replica: d4895b38e54f46428745ef398367cdc4, State: Running, Role: LEADER
I20260929 20:51:54.621600 21950 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4895b38e54f46428745ef398367cdc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33573 } }
I20260929 20:51:54.622364 21940 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:51:54.624258 21951 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4895b38e54f46428745ef398367cdc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4895b38e54f46428745ef398367cdc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33573 } } }
I20260929 20:51:54.624348 21952 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4895b38e54f46428745ef398367cdc4. Latest consensus state: current_term: 1 leader_uuid: "d4895b38e54f46428745ef398367cdc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4895b38e54f46428745ef398367cdc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33573 } } }
I20260929 20:51:54.624632 21951 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:54.624660 21952 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:54.632239 21955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:51:54.637447 21955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:51:54.645911 21955 catalog_manager.cc:1347] Generated new cluster ID: f84f588ced2446e38bee4b1f0fca5c38
I20260929 20:51:54.646051 21955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:51:54.660313 21955 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:51:54.661602 21955 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:51:54.673592 21955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4895b38e54f46428745ef398367cdc4: Generated new TSK 0
I20260929 20:51:54.674423 21955 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:51:54.687222 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 21873
W20260929 20:51:54.688472 21935 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.17:54793
W20260929 20:51:54.689204 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 21873: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:51:54.689352 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 21873
W20260929 20:51:54.702675 21858 connection.cc:535] client connection to 127.0.0.1:33573 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33573): Connection reset by peer (error 104)
2026-09-29T20:51:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1779 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:51:54 dist-test-slave-blv7 krb5kdc[21971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:51:54 dist-test-slave-blv7 krb5kdc[21971](info): set up 2 sockets
Sep 29 20:51:54 dist-test-slave-blv7 krb5kdc[21971](info): commencing operation
krb5kdc: starting...
W20260929 20:51:55.699081 388 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:51:55 dist-test-slave-blv7 krb5kdc[21971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:51:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:51:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:51:55.939504 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34655
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:56.264989 21984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:56.265314 21984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:56.265453 21984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:56.272725 21984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:51:56.272858 21984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:56.272936 21984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:56.273020 21984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:51:56.273080 21984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:51:56.283054 21984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34655
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:56.284354 21984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:56.285631 21984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:56.297034 21989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:56.297008 21990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:56.297056 21992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:56.297864 21984 server_base.cc:1034] running on GCE node
I20260929 20:51:56.298641 21984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:56.300086 21984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:56.301316 21984 hybrid_clock.cc:648] HybridClock initialized: now 1790715116301264 us; error 40 us; skew 500 ppm
I20260929 20:51:56.301687 21984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:56 dist-test-slave-blv7 krb5kdc[21971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:56.311493 21984 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:51:56.313225 21984 webserver.cc:460] Webserver started at http://127.0.0.1:38065/ using document root <none> and password file <none>
I20260929 20:51:56.313994 21984 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:56.314128 21984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:56.314458 21984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:56.317725 21984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "472f18009eac4261ae73c6ac3d9b8303"
format_stamp: "Formatted at 2026-09-29 20:51:56 on dist-test-slave-blv7"
server_key: "11f0dc4bf00c78c74b02070ca9ca2934"
server_key_iv: "e507bb922b63f1b5ee7186f2c783dda7"
server_key_version: "encryptionkey@0"
I20260929 20:51:56.318568 21984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "472f18009eac4261ae73c6ac3d9b8303"
format_stamp: "Formatted at 2026-09-29 20:51:56 on dist-test-slave-blv7"
server_key: "11f0dc4bf00c78c74b02070ca9ca2934"
server_key_iv: "e507bb922b63f1b5ee7186f2c783dda7"
server_key_version: "encryptionkey@0"
I20260929 20:51:56.324314 21984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 20:51:56.328058 21999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:56.329397 21984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:51:56.329567 21984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "472f18009eac4261ae73c6ac3d9b8303"
format_stamp: "Formatted at 2026-09-29 20:51:56 on dist-test-slave-blv7"
server_key: "11f0dc4bf00c78c74b02070ca9ca2934"
server_key_iv: "e507bb922b63f1b5ee7186f2c783dda7"
server_key_version: "encryptionkey@0"
I20260929 20:51:56.329751 21984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:56.335309 21984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:56.339365 21984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:56.339661 21984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:56.356818 21984 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34655
I20260929 20:51:56.356839 22050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34655 every 8 connection(s)
I20260929 20:51:56.358206 21984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:51:56.361929 22051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:56.362046 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 21984
I20260929 20:51:56.362221 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:51:56.362732 388 external_mini_cluster.cc:1428] Setting key 3bdaf661da2652ed61282d2683e0031e
I20260929 20:51:56.370169 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:56.364569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:54123 (local address 127.0.0.1:34655)
0929 20:51:56.365366 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:56.365381 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:56.365430 (+ 49us) server_negotiation.cc:407] Connection header received
0929 20:51:56.366874 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:56.366895 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:56.367283 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:56.367637 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:56.368635 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:56.368641 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:56.368655 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:56.368670 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:56.368740 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:56.368760 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:56.368969 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:56.369107 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 20:51:56.369285 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":337,"threads_started":1}
I20260929 20:51:56.373667 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:51:56.375571 22051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303: Bootstrap starting.
I20260929 20:51:56.379148 22051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:56.379978 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:56.377644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.17:37137 (local address 127.0.0.1:34655)
0929 20:51:56.377830 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:56.377836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:56.378199 (+ 363us) server_negotiation.cc:407] Connection header received
0929 20:51:56.378331 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:56.378337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:56.378404 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:56.378529 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:56.379370 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:56.379375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:56.379380 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:51:56.379399 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:56.379440 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:51:56.379459 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:56.379563 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:56.379665 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 20:51:56.379721 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:51:56.380434 22051 log.cc:824] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303: Log is configured to *not* fsync() on all Append() calls
I20260929 20:51:56.383343 22051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303: No bootstrap required, opened a new log
I20260929 20:51:56.390777 22051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472f18009eac4261ae73c6ac3d9b8303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34655 } }
I20260929 20:51:56.391220 22051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:56.391330 22051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 472f18009eac4261ae73c6ac3d9b8303, State: Initialized, Role: FOLLOWER
I20260929 20:51:56.391921 22051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472f18009eac4261ae73c6ac3d9b8303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34655 } }
I20260929 20:51:56.392155 22051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:51:56.392302 22051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:51:56.392464 22051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:56.394735 22051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472f18009eac4261ae73c6ac3d9b8303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34655 } }
I20260929 20:51:56.395319 22051 leader_election.cc:302] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 472f18009eac4261ae73c6ac3d9b8303; no voters:
I20260929 20:51:56.395906 22051 leader_election.cc:288] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:51:56.396021 22061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:56.396673 22061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303 [term 1 LEADER]: Becoming Leader. State: Replica: 472f18009eac4261ae73c6ac3d9b8303, State: Running, Role: LEADER
I20260929 20:51:56.397183 22061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472f18009eac4261ae73c6ac3d9b8303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34655 } }
I20260929 20:51:56.398093 22051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:51:56.399752 22062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "472f18009eac4261ae73c6ac3d9b8303" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472f18009eac4261ae73c6ac3d9b8303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34655 } } }
I20260929 20:51:56.400090 22062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:56.400913 22066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:51:56.400753 22063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 472f18009eac4261ae73c6ac3d9b8303. Latest consensus state: current_term: 1 leader_uuid: "472f18009eac4261ae73c6ac3d9b8303" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472f18009eac4261ae73c6ac3d9b8303" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34655 } } }
I20260929 20:51:56.401054 22063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:56.405259 22066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:51:56.415222 22066 catalog_manager.cc:1347] Generated new cluster ID: ff59bdcdbccd438eb435c5ab675e4f21
I20260929 20:51:56.415335 22066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:51:56.443717 22066 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:51:56.444284 22066 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:51:56.452245 22066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 472f18009eac4261ae73c6ac3d9b8303: Generated new TSK 0
I20260929 20:51:56.453357 22066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:51:56.462832 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 21984
W20260929 20:51:56.464246 22046 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.17:54123
W20260929 20:51:56.464927 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 21984: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:51:56.465049 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 21984
2026-09-29T20:51:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1775 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:51:56 dist-test-slave-blv7 krb5kdc[22082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:51:56 dist-test-slave-blv7 krb5kdc[22082](info): set up 2 sockets
Sep 29 20:51:56 dist-test-slave-blv7 krb5kdc[22082](info): commencing operation
krb5kdc: starting...
W20260929 20:51:57.479218 388 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.009s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:51:57 dist-test-slave-blv7 krb5kdc[22082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:51:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:51:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:51:57.721550 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35649
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:58.040380 22095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:58.040704 22095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:58.040846 22095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:58.048192 22095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:51:58.048313 22095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:58.048388 22095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:58.048468 22095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:51:58.048528 22095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:51:58.058374 22095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35649
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:58.059674 22095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:58.060971 22095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:58.071940 22101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:58.072057 22100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:58.072058 22103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:58.073133 22095 server_base.cc:1034] running on GCE node
I20260929 20:51:58.073868 22095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:58.075314 22095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:58.076529 22095 hybrid_clock.cc:648] HybridClock initialized: now 1790715118076476 us; error 46 us; skew 500 ppm
I20260929 20:51:58.076870 22095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:58 dist-test-slave-blv7 krb5kdc[22082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:58.080009 22095 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:51:58.081566 22095 webserver.cc:460] Webserver started at http://127.0.97.62:41515/ using document root <none> and password file <none>
I20260929 20:51:58.082329 22095 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:58.082527 22095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:58.082859 22095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:58.085882 22095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "50f38ba24cde4e75afb5cc0184ffa6d4"
format_stamp: "Formatted at 2026-09-29 20:51:58 on dist-test-slave-blv7"
server_key: "6b7b37a785ad45ff0eb6b15547a76d84"
server_key_iv: "81153bbe26e8667af71d414be06fd1ec"
server_key_version: "encryptionkey@0"
I20260929 20:51:58.086627 22095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "50f38ba24cde4e75afb5cc0184ffa6d4"
format_stamp: "Formatted at 2026-09-29 20:51:58 on dist-test-slave-blv7"
server_key: "6b7b37a785ad45ff0eb6b15547a76d84"
server_key_iv: "81153bbe26e8667af71d414be06fd1ec"
server_key_version: "encryptionkey@0"
I20260929 20:51:58.091646 22095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:51:58.095204 22110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:58.096596 22095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:51:58.096746 22095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "50f38ba24cde4e75afb5cc0184ffa6d4"
format_stamp: "Formatted at 2026-09-29 20:51:58 on dist-test-slave-blv7"
server_key: "6b7b37a785ad45ff0eb6b15547a76d84"
server_key_iv: "81153bbe26e8667af71d414be06fd1ec"
server_key_version: "encryptionkey@0"
I20260929 20:51:58.096935 22095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:58.115592 22095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:58.119742 22095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:58.120014 22095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:58.136171 22095 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35649
I20260929 20:51:58.136194 22161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35649 every 8 connection(s)
I20260929 20:51:58.137549 22095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:51:58.141251 22162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:58.143646 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 22095
I20260929 20:51:58.143800 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:51:58.144204 388 external_mini_cluster.cc:1428] Setting key 41511d8daf876fd5249c9b7f6d8d47ae
I20260929 20:51:58.154070 22162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4: Bootstrap starting.
I20260929 20:51:58.157583 22162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4: Neither blocks nor log segments found. Creating new log.
I20260929 20:51:58.158574 22162 log.cc:824] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:51:58 dist-test-slave-blv7 krb5kdc[22082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:51:58.161412 22162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4: No bootstrap required, opened a new log
I20260929 20:51:58.168426 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:58.145998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36133 (local address 127.0.97.62:35649)
0929 20:51:58.146825 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:58.146843 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:58.146893 (+ 50us) server_negotiation.cc:407] Connection header received
0929 20:51:58.149200 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:58.149230 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:58.149758 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:58.150237 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:58.151436 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:58.153361 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:58.154678 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:58.155728 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:58.155935 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:58.159925 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:58.159933 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:58.159950 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:58.159988 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:58.163562 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:58.164396 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:58.164405 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:58.164414 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:58.164661 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:58.165089 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:58.165094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:58.165098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:58.165850 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:58.166156 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:58.166590 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 20:51:58.166950 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":332,"threads_started":1}
I20260929 20:51:58.171545 22162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f38ba24cde4e75afb5cc0184ffa6d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35649 } }
I20260929 20:51:58.171972 22162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:51:58.172071 22162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50f38ba24cde4e75afb5cc0184ffa6d4, State: Initialized, Role: FOLLOWER
I20260929 20:51:58.172456 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:51:58.172701 22162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f38ba24cde4e75afb5cc0184ffa6d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35649 } }
I20260929 20:51:58.172960 22162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:51:58.173074 22162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:51:58.173219 22162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:51:58.176340 22162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f38ba24cde4e75afb5cc0184ffa6d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35649 } }
I20260929 20:51:58.176970 22162 leader_election.cc:302] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50f38ba24cde4e75afb5cc0184ffa6d4; no voters:
I20260929 20:51:58.177604 22162 leader_election.cc:288] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:51:58.177699 22171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:51:58.178289 22171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4 [term 1 LEADER]: Becoming Leader. State: Replica: 50f38ba24cde4e75afb5cc0184ffa6d4, State: Running, Role: LEADER
I20260929 20:51:58.178990 22171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f38ba24cde4e75afb5cc0184ffa6d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35649 } }
I20260929 20:51:58.181075 22171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50f38ba24cde4e75afb5cc0184ffa6d4. Latest consensus state: current_term: 1 leader_uuid: "50f38ba24cde4e75afb5cc0184ffa6d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f38ba24cde4e75afb5cc0184ffa6d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35649 } } }
I20260929 20:51:58.182665 22171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:58.182307 22174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50f38ba24cde4e75afb5cc0184ffa6d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f38ba24cde4e75afb5cc0184ffa6d4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35649 } } }
I20260929 20:51:58.183172 22174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4 [sys.catalog]: This master's current role is: LEADER
I20260929 20:51:58.186169 22162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:51:58.186543 22176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:51:58.194152 22176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:51:58.194834 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:58.177419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33905 (local address 127.0.97.62:35649)
0929 20:51:58.177626 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:58.177632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:58.178136 (+ 504us) server_negotiation.cc:407] Connection header received
0929 20:51:58.178430 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:58.178437 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:58.178511 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:58.178580 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:58.180233 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:58.181295 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:58.183100 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:58.183758 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:51:58.183887 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:51:58.186415 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:51:58.186422 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:51:58.186427 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:51:58.186477 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:51:58.190385 (+ 3908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:58.191527 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:58.191534 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:58.191538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:58.191761 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:51:58.192364 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:51:58.192370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:51:58.192374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:51:58.192826 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:51:58.193100 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 20:51:58.194236 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0929 20:51:58.194439 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:51:58.204907 22176 catalog_manager.cc:1347] Generated new cluster ID: c0044abf0a4c4bf6991d91151c4deb84
I20260929 20:51:58.205030 22176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:51:58.221336 22176 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:51:58.222646 22176 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:51:58.229007 22176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50f38ba24cde4e75afb5cc0184ffa6d4: Generated new TSK 0
I20260929 20:51:58.230132 22176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:51:58.235857 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 22095
I20260929 20:51:58.236922 22157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:51:58.324604 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 22095
W20260929 20:51:58.337937 22080 connection.cc:535] client connection to 127.0.97.62:35649 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:35649): Connection reset by peer (error 104)
2026-09-29T20:51:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1859 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:51:58 dist-test-slave-blv7 krb5kdc[22194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:51:58 dist-test-slave-blv7 krb5kdc[22194](info): set up 2 sockets
Sep 29 20:51:58 dist-test-slave-blv7 krb5kdc[22194](info): commencing operation
krb5kdc: starting...
W20260929 20:51:59.335171 388 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:51:59 dist-test-slave-blv7 krb5kdc[22194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:51:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:51:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:51:59.569844 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37489
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:51:59.892309 22207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:51:59.892620 22207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:51:59.892762 22207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:51:59.900118 22207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:51:59.900243 22207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:51:59.900321 22207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:51:59.900405 22207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:51:59.900485 22207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:51:59.910460 22207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37489
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:51:59.911892 22207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:51:59.913173 22207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:51:59.924957 22212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:59.924980 22213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:51:59.924902 22215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:51:59.925880 22207 server_base.cc:1034] running on GCE node
I20260929 20:51:59.926653 22207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:51:59.928141 22207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:51:59.929368 22207 hybrid_clock.cc:648] HybridClock initialized: now 1790715119929308 us; error 51 us; skew 500 ppm
I20260929 20:51:59.929710 22207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:51:59 dist-test-slave-blv7 krb5kdc[22194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:51:59.932747 22207 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:51:59.934235 22207 webserver.cc:460] Webserver started at http://127.0.97.62:36023/ using document root <none> and password file <none>
I20260929 20:51:59.934908 22207 fs_manager.cc:359] Metadata directory not provided
I20260929 20:51:59.935048 22207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:51:59.935339 22207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:51:59.938269 22207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "c73c4207eaa749afb0bdb018f4c8743e"
format_stamp: "Formatted at 2026-09-29 20:51:59 on dist-test-slave-blv7"
server_key: "387f8ca7f561e97e5f0616fcb4230cd7"
server_key_iv: "78b27fd27b2c3183100f1b4f41882b68"
server_key_version: "encryptionkey@0"
I20260929 20:51:59.939004 22207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "c73c4207eaa749afb0bdb018f4c8743e"
format_stamp: "Formatted at 2026-09-29 20:51:59 on dist-test-slave-blv7"
server_key: "387f8ca7f561e97e5f0616fcb4230cd7"
server_key_iv: "78b27fd27b2c3183100f1b4f41882b68"
server_key_version: "encryptionkey@0"
I20260929 20:51:59.944073 22207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:51:59.947432 22222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:51:59.948690 22207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:51:59.948849 22207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "c73c4207eaa749afb0bdb018f4c8743e"
format_stamp: "Formatted at 2026-09-29 20:51:59 on dist-test-slave-blv7"
server_key: "387f8ca7f561e97e5f0616fcb4230cd7"
server_key_iv: "78b27fd27b2c3183100f1b4f41882b68"
server_key_version: "encryptionkey@0"
I20260929 20:51:59.949029 22207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:51:59.960508 22207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:51:59.964285 22207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:51:59.964546 22207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:51:59.981302 22207 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37489
I20260929 20:51:59.981312 22273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37489 every 8 connection(s)
I20260929 20:51:59.982654 22207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:51:59.986245 22274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:51:59.992156 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 22207
I20260929 20:51:59.992332 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:51:59.992852 388 external_mini_cluster.cc:1428] Setting key 1255a68ddf4bc354752c3cd69e0926fd
I20260929 20:51:59.997273 22274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e: Bootstrap starting.
I20260929 20:52:00.000478 22274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e: Neither blocks nor log segments found. Creating new log.
I20260929 20:52:00.001510 22274 log.cc:824] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:52:00.004295 22274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e: No bootstrap required, opened a new log
I20260929 20:52:00.005591 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:51:59.994706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46681 (local address 127.0.97.62:37489)
0929 20:51:59.995504 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0929 20:51:59.995516 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:51:59.995561 (+ 45us) server_negotiation.cc:407] Connection header received
0929 20:51:59.997185 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:51:59.997207 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:51:59.997648 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:51:59.998044 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:51:59.998568 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:51:59.999902 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:00.001272 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:00.002027 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:00.002213 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:00.003402 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:00.003409 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:00.003423 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:52:00.003958 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:00.004029 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:52:00.004058 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:00.004195 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:00.004493 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 20:52:00.004647 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":347,"threads_started":1}
I20260929 20:52:00.009358 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:52:00.012084 22274 raft_consensus.cc:348] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73c4207eaa749afb0bdb018f4c8743e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37489 } }
I20260929 20:52:00.012486 22274 raft_consensus.cc:374] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:52:00.012590 22274 raft_consensus.cc:729] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c73c4207eaa749afb0bdb018f4c8743e, State: Initialized, Role: FOLLOWER
I20260929 20:52:00.013216 22274 consensus_queue.cc:260] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73c4207eaa749afb0bdb018f4c8743e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37489 } }
I20260929 20:52:00.013526 22274 raft_consensus.cc:388] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:52:00.013670 22274 raft_consensus.cc:482] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:52:00.013854 22274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:52:00.016714 22274 raft_consensus.cc:504] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73c4207eaa749afb0bdb018f4c8743e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37489 } }
I20260929 20:52:00.017199 22274 leader_election.cc:302] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c73c4207eaa749afb0bdb018f4c8743e; no voters:
I20260929 20:52:00.017833 22274 leader_election.cc:288] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:52:00.017947 22284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:52:00.019551 22284 raft_consensus.cc:686] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e [term 1 LEADER]: Becoming Leader. State: Replica: c73c4207eaa749afb0bdb018f4c8743e, State: Running, Role: LEADER
I20260929 20:52:00.020170 22284 consensus_queue.cc:237] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73c4207eaa749afb0bdb018f4c8743e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37489 } }
I20260929 20:52:00.020701 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:00.013438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40627 (local address 127.0.97.62:37489)
0929 20:52:00.013586 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:00.013590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:00.014160 (+ 570us) server_negotiation.cc:407] Connection header received
0929 20:52:00.014305 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:00.014311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:00.014402 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:00.014517 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:00.014931 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:00.016037 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:00.017335 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:00.018043 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:00.018189 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:00.019469 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:00.019475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:00.019480 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:52:00.019504 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:00.019581 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:52:00.019606 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:00.020241 (+ 635us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:00.020322 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 20:52:00.020373 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 20:52:00.020762 22274 sys_catalog.cc:564] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:52:00.022822 22285 sys_catalog.cc:455] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c73c4207eaa749afb0bdb018f4c8743e. Latest consensus state: current_term: 1 leader_uuid: "c73c4207eaa749afb0bdb018f4c8743e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73c4207eaa749afb0bdb018f4c8743e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37489 } } }
I20260929 20:52:00.022948 22286 sys_catalog.cc:455] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c73c4207eaa749afb0bdb018f4c8743e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c73c4207eaa749afb0bdb018f4c8743e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37489 } } }
I20260929 20:52:00.023322 22285 sys_catalog.cc:458] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e [sys.catalog]: This master's current role is: LEADER
I20260929 20:52:00.023350 22286 sys_catalog.cc:458] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e [sys.catalog]: This master's current role is: LEADER
I20260929 20:52:00.028210 22292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:52:00.033197 22292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:52:00.042296 22292 catalog_manager.cc:1347] Generated new cluster ID: 965a5dfa0e014fdfbea622af90cb4f53
I20260929 20:52:00.042428 22292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:52:00.058828 22292 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:52:00.060169 22292 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:52:00.065073 22292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c73c4207eaa749afb0bdb018f4c8743e: Generated new TSK 0
I20260929 20:52:00.065806 22292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:52:00.071758 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 22207
W20260929 20:52:00.073194 22269 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46681
W20260929 20:52:00.073951 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 22207: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:52:00.074076 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 22207
W20260929 20:52:00.087467 22192 connection.cc:535] client connection to 127.0.97.62:37489 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:37489): Connection reset by peer (error 104)
2026-09-29T20:52:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1749 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:52:00 dist-test-slave-blv7 krb5kdc[22305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:52:00 dist-test-slave-blv7 krb5kdc[22305](info): set up 2 sockets
Sep 29 20:52:00 dist-test-slave-blv7 krb5kdc[22305](info): commencing operation
krb5kdc: starting...
W20260929 20:52:01.083204 388 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:52:01 dist-test-slave-blv7 krb5kdc[22305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:52:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:52:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:52:01.322321 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34061
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:52:01.640792 22318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:52:01.641072 22318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:52:01.641161 22318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:52:01.648350 22318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:52:01.648471 22318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:52:01.648514 22318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:52:01.648574 22318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:52:01.648615 22318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:52:01.658450 22318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34061
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:52:01.659781 22318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:52:01.661047 22318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:52:01.673107 22324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:01.673341 22326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:01.673514 22323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:52:01.674270 22318 server_base.cc:1034] running on GCE node
I20260929 20:52:01.675216 22318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:52:01.676669 22318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:52:01.677870 22318 hybrid_clock.cc:648] HybridClock initialized: now 1790715121677821 us; error 39 us; skew 500 ppm
I20260929 20:52:01.678200 22318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:52:01 dist-test-slave-blv7 krb5kdc[22305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:52:01.701427 22318 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:52:01.703128 22318 webserver.cc:460] Webserver started at http://127.0.97.62:35485/ using document root <none> and password file <none>
I20260929 20:52:01.703821 22318 fs_manager.cc:359] Metadata directory not provided
I20260929 20:52:01.703938 22318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:52:01.704252 22318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:52:01.707388 22318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "3122221f0e614846bf483481bba58813"
format_stamp: "Formatted at 2026-09-29 20:52:01 on dist-test-slave-blv7"
server_key: "e0887d96f23e00bc73b488b39e7077a1"
server_key_iv: "fbc2c02e70d6085d149da76095409046"
server_key_version: "encryptionkey@0"
I20260929 20:52:01.708109 22318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "3122221f0e614846bf483481bba58813"
format_stamp: "Formatted at 2026-09-29 20:52:01 on dist-test-slave-blv7"
server_key: "e0887d96f23e00bc73b488b39e7077a1"
server_key_iv: "fbc2c02e70d6085d149da76095409046"
server_key_version: "encryptionkey@0"
I20260929 20:52:01.713172 22318 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:52:01.716964 22333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:01.718629 22318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:52:01.718817 22318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "3122221f0e614846bf483481bba58813"
format_stamp: "Formatted at 2026-09-29 20:52:01 on dist-test-slave-blv7"
server_key: "e0887d96f23e00bc73b488b39e7077a1"
server_key_iv: "fbc2c02e70d6085d149da76095409046"
server_key_version: "encryptionkey@0"
I20260929 20:52:01.719087 22318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:52:01.725277 22318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:52:01.729307 22318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:52:01.729557 22318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:52:01.745899 22318 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34061
I20260929 20:52:01.745936 22384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34061 every 8 connection(s)
I20260929 20:52:01.747292 22318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:52:01.750885 22385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:52:01.755183 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 22318
I20260929 20:52:01.755326 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:52:01.755730 388 external_mini_cluster.cc:1428] Setting key caa257bcd8142a96599ea299b45a5d8b
I20260929 20:52:01.762166 22385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813: Bootstrap starting.
I20260929 20:52:01.763664 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:01.757430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59221 (local address 127.0.97.62:34061)
0929 20:52:01.758180 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:01.758192 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:01.758230 (+ 38us) server_negotiation.cc:407] Connection header received
0929 20:52:01.760379 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:01.760416 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:01.760751 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:01.761113 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:01.762100 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:01.762107 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:01.762120 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:52:01.762135 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:01.762197 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:52:01.762218 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:01.762410 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:01.762537 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 20:52:01.762713 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":330,"threads_started":1}
I20260929 20:52:01.765681 22385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813: Neither blocks nor log segments found. Creating new log.
I20260929 20:52:01.766973 22385 log.cc:824] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813: Log is configured to *not* fsync() on all Append() calls
I20260929 20:52:01.767385 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:52:01.769690 22385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813: No bootstrap required, opened a new log
I20260929 20:52:01.773651 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:01.771311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44835 (local address 127.0.97.62:34061)
0929 20:52:01.771511 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:01.771518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:01.771874 (+ 356us) server_negotiation.cc:407] Connection header received
0929 20:52:01.772009 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:01.772015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:01.772077 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:01.772181 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:01.773008 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:01.773014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:01.773019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:52:01.773042 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:01.773093 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:52:01.773112 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:01.773252 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:01.773297 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 20:52:01.773359 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:52:01.778103 22385 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3122221f0e614846bf483481bba58813" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34061 } }
I20260929 20:52:01.778450 22385 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:52:01.778529 22385 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3122221f0e614846bf483481bba58813, State: Initialized, Role: FOLLOWER
I20260929 20:52:01.779070 22385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3122221f0e614846bf483481bba58813" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34061 } }
I20260929 20:52:01.779291 22385 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:52:01.779402 22385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:52:01.779552 22385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:52:01.781826 22385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3122221f0e614846bf483481bba58813" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34061 } }
I20260929 20:52:01.782303 22385 leader_election.cc:302] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3122221f0e614846bf483481bba58813; no voters:
I20260929 20:52:01.782891 22385 leader_election.cc:288] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:52:01.783008 22395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:52:01.783628 22395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813 [term 1 LEADER]: Becoming Leader. State: Replica: 3122221f0e614846bf483481bba58813, State: Running, Role: LEADER
I20260929 20:52:01.784515 22395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3122221f0e614846bf483481bba58813" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34061 } }
I20260929 20:52:01.785144 22385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:52:01.787344 22397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3122221f0e614846bf483481bba58813. Latest consensus state: current_term: 1 leader_uuid: "3122221f0e614846bf483481bba58813" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3122221f0e614846bf483481bba58813" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34061 } } }
I20260929 20:52:01.787755 22397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813 [sys.catalog]: This master's current role is: LEADER
I20260929 20:52:01.788192 22396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3122221f0e614846bf483481bba58813" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3122221f0e614846bf483481bba58813" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34061 } } }
I20260929 20:52:01.788482 22396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813 [sys.catalog]: This master's current role is: LEADER
I20260929 20:52:01.788753 22403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:52:01.794395 22403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:52:01.802117 22403 catalog_manager.cc:1347] Generated new cluster ID: b652184584f6410eb76f5d5fd812f44e
I20260929 20:52:01.802285 22403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:52:01.825729 22403 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:52:01.826251 22403 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:52:01.832892 22403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3122221f0e614846bf483481bba58813: Generated new TSK 0
I20260929 20:52:01.833994 22403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:52:01.841949 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 22318
W20260929 20:52:01.843068 22380 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59221
W20260929 20:52:01.843698 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 22318: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:52:01.843815 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 22318
2026-09-29T20:52:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1769 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:52:01 dist-test-slave-blv7 krb5kdc[22416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:52:01 dist-test-slave-blv7 krb5kdc[22416](info): set up 2 sockets
Sep 29 20:52:01 dist-test-slave-blv7 krb5kdc[22416](info): commencing operation
krb5kdc: starting...
W20260929 20:52:02.855187 388 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:52:03 dist-test-slave-blv7 krb5kdc[22416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:52:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:52:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:52:03.093787 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34203
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:52:03.417294 22429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:52:03.417598 22429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:52:03.417728 22429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:52:03.425186 22429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:52:03.425303 22429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:52:03.425372 22429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:52:03.425450 22429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:52:03.425509 22429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:52:03.435638 22429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:52:03.436972 22429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:52:03.438285 22429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:52:03.449661 22434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:03.449705 22435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:03.449728 22437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:52:03.450855 22429 server_base.cc:1034] running on GCE node
I20260929 20:52:03.451723 22429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:52:03.453199 22429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:52:03.454416 22429 hybrid_clock.cc:648] HybridClock initialized: now 1790715123454366 us; error 42 us; skew 500 ppm
I20260929 20:52:03.454761 22429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:52:03 dist-test-slave-blv7 krb5kdc[22416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:52:03.457811 22429 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:52:03.459386 22429 webserver.cc:460] Webserver started at http://127.0.97.62:44209/ using document root <none> and password file <none>
I20260929 20:52:03.460052 22429 fs_manager.cc:359] Metadata directory not provided
I20260929 20:52:03.460151 22429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:52:03.460443 22429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:52:03.463529 22429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "b0098482f60948bf95eeabeda0a6fc41"
format_stamp: "Formatted at 2026-09-29 20:52:03 on dist-test-slave-blv7"
server_key: "b4e52d49ec3d572344e4ce338c445155"
server_key_iv: "7099edb30c0b2fc79d9d9556f9679ab1"
server_key_version: "encryptionkey@0"
I20260929 20:52:03.464270 22429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "b0098482f60948bf95eeabeda0a6fc41"
format_stamp: "Formatted at 2026-09-29 20:52:03 on dist-test-slave-blv7"
server_key: "b4e52d49ec3d572344e4ce338c445155"
server_key_iv: "7099edb30c0b2fc79d9d9556f9679ab1"
server_key_version: "encryptionkey@0"
I20260929 20:52:03.469512 22429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:52:03.473438 22444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:03.474897 22429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:52:03.475096 22429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "b0098482f60948bf95eeabeda0a6fc41"
format_stamp: "Formatted at 2026-09-29 20:52:03 on dist-test-slave-blv7"
server_key: "b4e52d49ec3d572344e4ce338c445155"
server_key_iv: "7099edb30c0b2fc79d9d9556f9679ab1"
server_key_version: "encryptionkey@0"
I20260929 20:52:03.475294 22429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:52:03.502655 22429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:52:03.506881 22429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:52:03.507169 22429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:52:03.524503 22429 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34203
I20260929 20:52:03.524533 22495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34203 every 8 connection(s)
I20260929 20:52:03.525826 22429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:52:03.526867 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 22429
I20260929 20:52:03.527056 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:52:03.527562 388 external_mini_cluster.cc:1428] Setting key 9ecf0763c6177d096ecee419a66e7b7f
I20260929 20:52:03.530231 22496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:52:03.540445 22496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41: Bootstrap starting.
Sep 29 20:52:03 dist-test-slave-blv7 krb5kdc[22416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:52:03.543767 22496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41: Neither blocks nor log segments found. Creating new log.
I20260929 20:52:03.544783 22496 log.cc:824] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41: Log is configured to *not* fsync() on all Append() calls
I20260929 20:52:03.548034 22496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41: No bootstrap required, opened a new log
I20260929 20:52:03.549080 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:03.529336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35829 (local address 127.0.97.62:34203)
0929 20:52:03.530106 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:03.530117 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:03.530158 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:52:03.531912 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:03.531935 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:03.532311 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:03.532689 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:03.534071 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:03.535447 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:03.536700 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:03.537364 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:03.537537 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:03.541564 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:03.541572 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:03.541586 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:03.541622 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:03.544733 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:03.545486 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:03.545492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:03.545499 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:03.545661 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:03.546058 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:03.546067 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:03.546070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:03.546717 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:03.547003 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:03.547337 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 20:52:03.547758 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":352,"threads_started":1}
I20260929 20:52:03.552700 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:52:03.556135 22496 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0098482f60948bf95eeabeda0a6fc41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34203 } }
I20260929 20:52:03.556550 22496 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:52:03.556670 22496 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0098482f60948bf95eeabeda0a6fc41, State: Initialized, Role: FOLLOWER
I20260929 20:52:03.557337 22496 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0098482f60948bf95eeabeda0a6fc41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34203 } }
I20260929 20:52:03.557613 22496 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:52:03.557760 22496 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:52:03.557940 22496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:52:03.560671 22496 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0098482f60948bf95eeabeda0a6fc41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34203 } }
I20260929 20:52:03.561220 22496 leader_election.cc:302] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0098482f60948bf95eeabeda0a6fc41; no voters:
I20260929 20:52:03.561839 22496 leader_election.cc:288] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:52:03.561957 22506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:52:03.563249 22506 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41 [term 1 LEADER]: Becoming Leader. State: Replica: b0098482f60948bf95eeabeda0a6fc41, State: Running, Role: LEADER
I20260929 20:52:03.563783 22506 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0098482f60948bf95eeabeda0a6fc41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34203 } }
I20260929 20:52:03.564013 22496 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:52:03.572741 22508 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0098482f60948bf95eeabeda0a6fc41. Latest consensus state: current_term: 1 leader_uuid: "b0098482f60948bf95eeabeda0a6fc41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0098482f60948bf95eeabeda0a6fc41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34203 } } }
I20260929 20:52:03.573165 22508 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41 [sys.catalog]: This master's current role is: LEADER
I20260929 20:52:03.573625 22507 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0098482f60948bf95eeabeda0a6fc41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0098482f60948bf95eeabeda0a6fc41" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34203 } } }
I20260929 20:52:03.573855 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:03.556662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45819 (local address 127.0.97.62:34203)
0929 20:52:03.556847 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:03.556854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:03.557311 (+ 457us) server_negotiation.cc:407] Connection header received
0929 20:52:03.557482 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:03.557486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:03.557546 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:03.557640 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:03.559388 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:03.560109 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:03.561800 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:03.562527 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:03.562652 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:03.566018 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:03.566025 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:03.566031 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:03.566072 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:03.570012 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:03.570695 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:03.570700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:03.570704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:03.570913 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:03.571287 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:03.571294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:03.571296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:03.571692 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:03.571995 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:03.573218 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0929 20:52:03.573461 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 20:52:03.573891 22507 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41 [sys.catalog]: This master's current role is: LEADER
W20260929 20:52:03.580813 22520 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:52:03.580937 22520 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:52:03.581069 22521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:52:03.583045 22521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:52:03.591760 22521 catalog_manager.cc:1347] Generated new cluster ID: c704fae22d8d44418e4d38282f9faa2e
I20260929 20:52:03.591869 22521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:52:03.635767 22521 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:52:03.637097 22521 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:52:03.649964 22521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0098482f60948bf95eeabeda0a6fc41: Generated new TSK 0
I20260929 20:52:03.650843 22521 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:52:03.659981 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 22429
I20260929 20:52:03.661108 22491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:52:03.755756 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 22429
2026-09-29T20:52:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1913 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:52:03 dist-test-slave-blv7 krb5kdc[22528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:52:03 dist-test-slave-blv7 krb5kdc[22528](info): set up 2 sockets
Sep 29 20:52:03 dist-test-slave-blv7 krb5kdc[22528](info): commencing operation
krb5kdc: starting...
W20260929 20:52:04.779050 388 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:52:04 dist-test-slave-blv7 krb5kdc[22528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:52:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:52:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:52:05.011711 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33571
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:52:05.326082 22541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:52:05.326399 22541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:52:05.326529 22541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:52:05.333770 22541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:52:05.333889 22541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:52:05.333968 22541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:52:05.334043 22541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:52:05.334100 22541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:52:05.343827 22541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:52:05.345027 22541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:52:05.346289 22541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:52:05.357892 22546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:05.357833 22549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:05.357901 22547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:52:05.358616 22541 server_base.cc:1034] running on GCE node
I20260929 20:52:05.359378 22541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:52:05.360675 22541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:52:05.361865 22541 hybrid_clock.cc:648] HybridClock initialized: now 1790715125361819 us; error 41 us; skew 500 ppm
I20260929 20:52:05.362181 22541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:52:05 dist-test-slave-blv7 krb5kdc[22528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:52:05.365054 22541 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:52:05.366453 22541 webserver.cc:460] Webserver started at http://127.0.97.62:37957/ using document root <none> and password file <none>
I20260929 20:52:05.367139 22541 fs_manager.cc:359] Metadata directory not provided
I20260929 20:52:05.367257 22541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:52:05.367542 22541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:52:05.370373 22541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "585384dec4644049b879f92e69ab0023"
format_stamp: "Formatted at 2026-09-29 20:52:05 on dist-test-slave-blv7"
server_key: "78ed7fe7bcef9afb92b8612b8e1e57c4"
server_key_iv: "66a3e319bcfe4e79d250e0bfa9dae962"
server_key_version: "encryptionkey@0"
I20260929 20:52:05.371124 22541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "585384dec4644049b879f92e69ab0023"
format_stamp: "Formatted at 2026-09-29 20:52:05 on dist-test-slave-blv7"
server_key: "78ed7fe7bcef9afb92b8612b8e1e57c4"
server_key_iv: "66a3e319bcfe4e79d250e0bfa9dae962"
server_key_version: "encryptionkey@0"
I20260929 20:52:05.376056 22541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 20:52:05.379242 22556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:05.380487 22541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:52:05.380637 22541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "585384dec4644049b879f92e69ab0023"
format_stamp: "Formatted at 2026-09-29 20:52:05 on dist-test-slave-blv7"
server_key: "78ed7fe7bcef9afb92b8612b8e1e57c4"
server_key_iv: "66a3e319bcfe4e79d250e0bfa9dae962"
server_key_version: "encryptionkey@0"
I20260929 20:52:05.380800 22541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:52:05.429847 22541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:52:05.433887 22541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:52:05.434149 22541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:52:05.451063 22541 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33571
I20260929 20:52:05.451084 22607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33571 every 8 connection(s)
I20260929 20:52:05.452435 22541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:52:05.453771 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 22541
I20260929 20:52:05.453953 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:52:05.454452 388 external_mini_cluster.cc:1428] Setting key 52c755cd96c5b0d1b8924b01a4347dee
I20260929 20:52:05.457003 22608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:52:05.467046 22611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:05.456258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36327 (local address 127.0.97.62:33571)
0929 20:52:05.456972 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:05.456984 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:05.457028 (+ 44us) server_negotiation.cc:407] Connection header received
0929 20:52:05.458670 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:05.458692 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:05.459128 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:05.459532 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:05.460068 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:05.461413 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:05.462714 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:05.463413 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:05.463575 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:05.464778 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:05.464785 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:05.464803 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:52:05.465365 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:05.465450 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:52:05.465479 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:05.465581 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:05.465822 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 20:52:05.466074 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":316,"threads_started":1}
I20260929 20:52:05.467720 22608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023: Bootstrap starting.
I20260929 20:52:05.470700 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:52:05.471210 22608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023: Neither blocks nor log segments found. Creating new log.
I20260929 20:52:05.472129 22608 log.cc:824] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023: Log is configured to *not* fsync() on all Append() calls
I20260929 20:52:05.475157 22608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023: No bootstrap required, opened a new log
I20260929 20:52:05.480542 22611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:05.474892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43751 (local address 127.0.97.62:33571)
0929 20:52:05.475046 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:05.475052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:05.475482 (+ 430us) server_negotiation.cc:407] Connection header received
0929 20:52:05.475649 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:05.475653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:05.475713 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:05.475790 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:05.476293 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:05.477015 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:05.478272 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:05.478688 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:05.478806 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:05.479872 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:05.479877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:05.479880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:52:05.479899 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:05.479934 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:52:05.479952 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:05.480033 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:05.480164 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 20:52:05.480252 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 20:52:05.482982 22608 raft_consensus.cc:348] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "585384dec4644049b879f92e69ab0023" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33571 } }
I20260929 20:52:05.483388 22608 raft_consensus.cc:374] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:52:05.483472 22608 raft_consensus.cc:729] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 585384dec4644049b879f92e69ab0023, State: Initialized, Role: FOLLOWER
I20260929 20:52:05.484015 22608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "585384dec4644049b879f92e69ab0023" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33571 } }
I20260929 20:52:05.484227 22608 raft_consensus.cc:388] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:52:05.484344 22608 raft_consensus.cc:482] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:52:05.484496 22608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:52:05.486598 22608 raft_consensus.cc:504] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "585384dec4644049b879f92e69ab0023" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33571 } }
I20260929 20:52:05.487109 22608 leader_election.cc:302] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 585384dec4644049b879f92e69ab0023; no voters:
I20260929 20:52:05.487706 22608 leader_election.cc:288] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:52:05.487860 22618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:52:05.488422 22618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023 [term 1 LEADER]: Becoming Leader. State: Replica: 585384dec4644049b879f92e69ab0023, State: Running, Role: LEADER
I20260929 20:52:05.488996 22618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "585384dec4644049b879f92e69ab0023" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33571 } }
I20260929 20:52:05.489902 22608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:52:05.491743 22619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "585384dec4644049b879f92e69ab0023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "585384dec4644049b879f92e69ab0023" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33571 } } }
I20260929 20:52:05.492110 22619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023 [sys.catalog]: This master's current role is: LEADER
I20260929 20:52:05.492141 22620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 585384dec4644049b879f92e69ab0023. Latest consensus state: current_term: 1 leader_uuid: "585384dec4644049b879f92e69ab0023" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "585384dec4644049b879f92e69ab0023" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33571 } } }
I20260929 20:52:05.492452 22620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023 [sys.catalog]: This master's current role is: LEADER
I20260929 20:52:05.493705 22625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:52:05.498742 22625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:52:05.507751 22625 catalog_manager.cc:1347] Generated new cluster ID: 87f41c1cb6624a6dad3eb27e0a01257f
I20260929 20:52:05.507867 22625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:52:05.517970 22625 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:52:05.519294 22625 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:52:05.531952 22625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 585384dec4644049b879f92e69ab0023: Generated new TSK 0
I20260929 20:52:05.532742 22625 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:52:05.545463 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 22541
W20260929 20:52:05.546520 22603 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36327
W20260929 20:52:05.547191 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 22541: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:52:05.547329 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 22541
2026-09-29T20:52:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1789 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:52:05 dist-test-slave-blv7 krb5kdc[22639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:52:05 dist-test-slave-blv7 krb5kdc[22639](info): set up 2 sockets
Sep 29 20:52:05 dist-test-slave-blv7 krb5kdc[22639](info): commencing operation
krb5kdc: starting...
W20260929 20:52:06.559144 388 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:52:06 dist-test-slave-blv7 krb5kdc[22639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:52:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:52:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:52:06.801399 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46451
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:52:07.112787 22652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:52:07.113076 22652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:52:07.113217 22652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:52:07.120302 22652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:52:07.120447 22652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:52:07.120528 22652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:52:07.120612 22652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:52:07.120671 22652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:52:07.130602 22652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:52:07.131881 22652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:52:07.133157 22652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:52:07.144304 22657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:07.144357 22660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:07.144383 22658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:52:07.145083 22652 server_base.cc:1034] running on GCE node
I20260929 20:52:07.145875 22652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:52:07.147387 22652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:52:07.148614 22652 hybrid_clock.cc:648] HybridClock initialized: now 1790715127148551 us; error 50 us; skew 500 ppm
I20260929 20:52:07.148972 22652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:52:07 dist-test-slave-blv7 krb5kdc[22639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:52:07.157945 22652 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:52:07.159437 22652 webserver.cc:460] Webserver started at http://127.0.97.62:44299/ using document root <none> and password file <none>
I20260929 20:52:07.160072 22652 fs_manager.cc:359] Metadata directory not provided
I20260929 20:52:07.160166 22652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:52:07.160431 22652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:52:07.163398 22652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "1a3b2624206b4cd09e1271ad6986708d"
format_stamp: "Formatted at 2026-09-29 20:52:07 on dist-test-slave-blv7"
server_key: "0b8e0a8ec2d995dd49e2f533efb5a26b"
server_key_iv: "0e8b973d10d87760cfb1a7c2712f30bf"
server_key_version: "encryptionkey@0"
I20260929 20:52:07.164086 22652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "1a3b2624206b4cd09e1271ad6986708d"
format_stamp: "Formatted at 2026-09-29 20:52:07 on dist-test-slave-blv7"
server_key: "0b8e0a8ec2d995dd49e2f533efb5a26b"
server_key_iv: "0e8b973d10d87760cfb1a7c2712f30bf"
server_key_version: "encryptionkey@0"
I20260929 20:52:07.168999 22652 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:52:07.172286 22667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:07.173588 22652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:52:07.173731 22652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "1a3b2624206b4cd09e1271ad6986708d"
format_stamp: "Formatted at 2026-09-29 20:52:07 on dist-test-slave-blv7"
server_key: "0b8e0a8ec2d995dd49e2f533efb5a26b"
server_key_iv: "0e8b973d10d87760cfb1a7c2712f30bf"
server_key_version: "encryptionkey@0"
I20260929 20:52:07.173904 22652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:52:07.179239 22652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:52:07.182952 22652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:52:07.183230 22652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:52:07.199874 22652 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46451
I20260929 20:52:07.199950 22718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46451 every 8 connection(s)
I20260929 20:52:07.201180 22652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:52:07.202656 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 22652
I20260929 20:52:07.202821 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:52:07.203290 388 external_mini_cluster.cc:1428] Setting key 21a420a4e8f3bff763c8df19c59f8841
I20260929 20:52:07.205220 22719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:52:07.211674 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:07.204991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54057 (local address 127.0.97.62:46451)
0929 20:52:07.205745 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:07.205760 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:07.205808 (+ 48us) server_negotiation.cc:407] Connection header received
0929 20:52:07.207898 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:07.207925 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:07.208366 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:07.208810 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:07.209785 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:07.209793 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:07.209811 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:52:07.209833 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:07.209920 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:52:07.209950 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:07.210156 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:07.210274 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 20:52:07.210462 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":305,"threads_started":1}
I20260929 20:52:07.215004 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:52:07.218650 22719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d: Bootstrap starting.
I20260929 20:52:07.222126 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:07.219741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44227 (local address 127.0.97.62:46451)
0929 20:52:07.219921 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:07.219927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:07.220380 (+ 453us) server_negotiation.cc:407] Connection header received
0929 20:52:07.220553 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:07.220556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:07.220606 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:07.220678 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:07.221541 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:07.221544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:07.221547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:52:07.221561 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:07.221597 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:52:07.221611 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:07.221695 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:07.221829 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 20:52:07.221871 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:52:07.222448 22719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d: Neither blocks nor log segments found. Creating new log.
I20260929 20:52:07.223567 22719 log.cc:824] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:52:07.226312 22719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d: No bootstrap required, opened a new log
I20260929 20:52:07.233757 22719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3b2624206b4cd09e1271ad6986708d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46451 } }
I20260929 20:52:07.234159 22719 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:52:07.234241 22719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a3b2624206b4cd09e1271ad6986708d, State: Initialized, Role: FOLLOWER
I20260929 20:52:07.234787 22719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3b2624206b4cd09e1271ad6986708d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46451 } }
I20260929 20:52:07.235006 22719 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:52:07.235188 22719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:52:07.235391 22719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:52:07.237578 22719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3b2624206b4cd09e1271ad6986708d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46451 } }
I20260929 20:52:07.238042 22719 leader_election.cc:302] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a3b2624206b4cd09e1271ad6986708d; no voters:
I20260929 20:52:07.238591 22719 leader_election.cc:288] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:52:07.238703 22729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:52:07.239349 22729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d [term 1 LEADER]: Becoming Leader. State: Replica: 1a3b2624206b4cd09e1271ad6986708d, State: Running, Role: LEADER
I20260929 20:52:07.239825 22729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3b2624206b4cd09e1271ad6986708d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46451 } }
I20260929 20:52:07.240554 22719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:52:07.242220 22731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a3b2624206b4cd09e1271ad6986708d. Latest consensus state: current_term: 1 leader_uuid: "1a3b2624206b4cd09e1271ad6986708d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3b2624206b4cd09e1271ad6986708d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46451 } } }
I20260929 20:52:07.242605 22731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d [sys.catalog]: This master's current role is: LEADER
I20260929 20:52:07.243283 22730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a3b2624206b4cd09e1271ad6986708d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a3b2624206b4cd09e1271ad6986708d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46451 } } }
I20260929 20:52:07.243633 22730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d [sys.catalog]: This master's current role is: LEADER
I20260929 20:52:07.246742 22734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:52:07.252067 22734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:52:07.260663 22734 catalog_manager.cc:1347] Generated new cluster ID: 691c0baf10044d309ae93751b40dd76b
I20260929 20:52:07.260779 22734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:52:07.281244 22734 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:52:07.281648 22734 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:52:07.296566 22734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a3b2624206b4cd09e1271ad6986708d: Generated new TSK 0
I20260929 20:52:07.297369 22734 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:52:07.308764 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 22652
W20260929 20:52:07.310151 22714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54057
W20260929 20:52:07.310940 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 22652: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:52:07.311086 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 22652
2026-09-29T20:52:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1764 ms)
[----------] 18 tests from AuthTokenIssuingTest (32226 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:52:07 dist-test-slave-blv7 krb5kdc[22750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:52:07 dist-test-slave-blv7 krb5kdc[22750](info): set up 2 sockets
Sep 29 20:52:07 dist-test-slave-blv7 krb5kdc[22750](info): commencing operation
krb5kdc: starting...
W20260929 20:52:08.315419 388 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:52:08 dist-test-slave-blv7 krb5kdc[22750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:52:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:52:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:52:08.559484 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39333
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39333,127.0.97.61:44563,127.0.97.60:34189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:52:08.874620 22763 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:52:08.874971 22763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:52:08.875087 22763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:52:08.875197 22763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:52:08.882304 22763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:52:08.882401 22763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:52:08.882481 22763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:52:08.882545 22763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:52:08.882628 22763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:52:08.892289 22763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39333,127.0.97.61:44563,127.0.97.60:34189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:52:08.893522 22763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:52:08.894785 22763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:52:08.905807 22768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:08.905841 22769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:08.905745 22771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:52:08.906121 22763 server_base.cc:1034] running on GCE node
I20260929 20:52:08.907054 22763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:52:08.908424 22763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:52:08.909637 22763 hybrid_clock.cc:648] HybridClock initialized: now 1790715128909579 us; error 43 us; skew 500 ppm
I20260929 20:52:08.909953 22763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:52:08 dist-test-slave-blv7 krb5kdc[22750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:52:08.912983 22763 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:52:08.914441 22763 webserver.cc:460] Webserver started at http://127.0.97.62:39509/ using document root <none> and password file <none>
I20260929 20:52:08.915169 22763 fs_manager.cc:359] Metadata directory not provided
I20260929 20:52:08.915269 22763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:52:08.915563 22763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:52:08.918476 22763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "7afe332fb7a04dd29a499c00fd678eb7"
format_stamp: "Formatted at 2026-09-29 20:52:08 on dist-test-slave-blv7"
server_key: "4f8f5cee3d9d1377915e1f8a3f81304e"
server_key_iv: "25b0448292f057a6838d281f46a5a736"
server_key_version: "encryptionkey@0"
I20260929 20:52:08.919322 22763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "7afe332fb7a04dd29a499c00fd678eb7"
format_stamp: "Formatted at 2026-09-29 20:52:08 on dist-test-slave-blv7"
server_key: "4f8f5cee3d9d1377915e1f8a3f81304e"
server_key_iv: "25b0448292f057a6838d281f46a5a736"
server_key_version: "encryptionkey@0"
I20260929 20:52:08.924470 22763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 20:52:08.928089 22778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:08.929507 22763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:52:08.929679 22763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "7afe332fb7a04dd29a499c00fd678eb7"
format_stamp: "Formatted at 2026-09-29 20:52:08 on dist-test-slave-blv7"
server_key: "4f8f5cee3d9d1377915e1f8a3f81304e"
server_key_iv: "25b0448292f057a6838d281f46a5a736"
server_key_version: "encryptionkey@0"
I20260929 20:52:08.929891 22763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:52:08.947227 22763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:52:08.951453 22763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:52:08.951696 22763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:52:08.968681 22763 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39333
I20260929 20:52:08.968688 22829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39333 every 8 connection(s)
I20260929 20:52:08.970023 22763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:52:08.970587 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 22763
I20260929 20:52:08.970779 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:52:08.971526 388 external_mini_cluster.cc:1428] Setting key 65a576c417b7395dbb7435a015ab1a64
I20260929 20:52:08.973834 22830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:52:08.980396 22830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 29 20:52:08 dist-test-slave-blv7 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:52:09.003670 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:08.983583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39333 (local address 127.0.0.1:37814)
0929 20:52:08.984480 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:08.984539 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:08.986861 (+ 2322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:08.987703 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:08.987714 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:08.988470 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:08.988941 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:08.988968 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:08.990521 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:08.990527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:08.992251 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:08.992281 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:08.993163 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:08.993168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:08.993303 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:08.993937 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:08.993960 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:08.996121 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:08.999873 (+ 3752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:08.999883 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:08.999893 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:09.000254 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:09.000698 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:09.000701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:09.000704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:09.000855 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:09.001718 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:09.001724 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:09.001936 (+ 212us) client_negotiation.cc:769] Sending connection context
0929 20:52:09.002138 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 20:52:09.002542 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":359,"threads_started":1}
I20260929 20:52:09.003692 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:08.983583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37814 (local address 127.0.97.62:39333)
0929 20:52:08.985367 (+ 1784us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:08.985374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:08.985418 (+ 44us) server_negotiation.cc:407] Connection header received
0929 20:52:08.987245 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:08.987251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:08.987484 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:08.987644 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:08.989147 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:08.990359 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:08.992396 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:08.992958 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:08.993083 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:08.996367 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:08.996377 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:08.996393 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:08.996437 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:08.999752 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:09.000416 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:09.000425 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:09.000428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:09.000600 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:09.000946 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:09.000950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:09.000953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:09.001603 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:09.001801 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:09.002319 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 20:52:09.002722 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":1588,"spinlock_wait_cycles":24320,"thread_start_us":1370,"threads_started":1}
I20260929 20:52:09.006520 22830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:52:09.007692 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.007263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44563 (local address 127.0.0.1:36120)
0929 20:52:09.007432 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.007554 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 20:52:09.007959 22782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44563: connect: Connection refused (error 111)
I20260929 20:52:09.009435 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.009050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44563 (local address 127.0.0.1:36132)
0929 20:52:09.009197 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.009284 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 20:52:09.009622 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44563: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44563: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 20:52:09.025745 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44563
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39333,127.0.97.61:44563,127.0.97.60:34189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:52:09.059008 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } attempt: 1
I20260929 20:52:09.060274 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.059768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44563 (local address 127.0.0.1:36142)
0929 20:52:09.059980 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.060101 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 20:52:09.061275 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.060907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44563 (local address 127.0.0.1:36152)
0929 20:52:09.061039 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.061128 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260929 20:52:09.061450 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44563: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44563: connect: Connection refused (error 111)
I20260929 20:52:09.129730 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } attempt: 2
I20260929 20:52:09.131127 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.130478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44563 (local address 127.0.0.1:36156)
0929 20:52:09.130765 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.130887 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 20:52:09.132311 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.131890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44563 (local address 127.0.0.1:36160)
0929 20:52:09.132044 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.132157 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 20:52:09.132541 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44563: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44563: connect: Connection refused (error 111)
I20260929 20:52:09.223819 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } attempt: 3
I20260929 20:52:09.225031 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.224528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44563 (local address 127.0.0.1:36174)
0929 20:52:09.224755 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.224867 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 20:52:09.226106 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.225674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44563 (local address 127.0.0.1:36184)
0929 20:52:09.225824 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.225915 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 20:52:09.226305 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44563: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44563: connect: Connection refused (error 111)
W20260929 20:52:09.344915 22837 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:52:09.345249 22837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:52:09.345350 22837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:52:09.345460 22837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:52:09.353263 22837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:52:09.353356 22837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:52:09.353427 22837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:52:09.353495 22837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:52:09.353582 22837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:52:09.363477 22837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39333,127.0.97.61:44563,127.0.97.60:34189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:52:09.364846 22837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:52:09.366132 22837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 20:52:09.369547 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } attempt: 4
I20260929 20:52:09.370779 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.370152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44563 (local address 127.0.0.1:36200)
0929 20:52:09.370492 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.370593 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260929 20:52:09.371912 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.371512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44563 (local address 127.0.0.1:36216)
0929 20:52:09.371654 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.371758 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 20:52:09.372216 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44563: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44563: connect: Connection refused (error 111)
W20260929 20:52:09.377560 22842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:09.377683 22843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:09.378223 22845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:52:09.378364 22837 server_base.cc:1034] running on GCE node
I20260929 20:52:09.379169 22837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:52:09.380462 22837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:52:09.381664 22837 hybrid_clock.cc:648] HybridClock initialized: now 1790715129381602 us; error 37 us; skew 500 ppm
I20260929 20:52:09.381966 22837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:52:09 dist-test-slave-blv7 krb5kdc[22750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:52:09.384938 22837 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 20:52:09.386307 22837 webserver.cc:460] Webserver started at http://127.0.97.61:34841/ using document root <none> and password file <none>
I20260929 20:52:09.386966 22837 fs_manager.cc:359] Metadata directory not provided
I20260929 20:52:09.387090 22837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:52:09.387395 22837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:52:09.390216 22837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-1/data/instance:
uuid: "ef7fec96f2a7441d90cc966b38b9b511"
format_stamp: "Formatted at 2026-09-29 20:52:09 on dist-test-slave-blv7"
server_key: "a7aae660b86379cb45fa1bb8dc909c6b"
server_key_iv: "44e01572860709c5c645df2e9d561583"
server_key_version: "encryptionkey@0"
I20260929 20:52:09.390947 22837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-1/wal/instance:
uuid: "ef7fec96f2a7441d90cc966b38b9b511"
format_stamp: "Formatted at 2026-09-29 20:52:09 on dist-test-slave-blv7"
server_key: "a7aae660b86379cb45fa1bb8dc909c6b"
server_key_iv: "44e01572860709c5c645df2e9d561583"
server_key_version: "encryptionkey@0"
I20260929 20:52:09.395831 22837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:52:09.399097 22852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:09.400434 22837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:52:09.400584 22837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-1/wal
uuid: "ef7fec96f2a7441d90cc966b38b9b511"
format_stamp: "Formatted at 2026-09-29 20:52:09 on dist-test-slave-blv7"
server_key: "a7aae660b86379cb45fa1bb8dc909c6b"
server_key_iv: "44e01572860709c5c645df2e9d561583"
server_key_version: "encryptionkey@0"
I20260929 20:52:09.400772 22837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:52:09.431224 22837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:52:09.435410 22837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:52:09.435653 22837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:52:09.452600 22837 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44563
I20260929 20:52:09.452606 22903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44563 every 8 connection(s)
I20260929 20:52:09.453970 22837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-1/data/info.pb
I20260929 20:52:09.457616 22904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:52:09.458887 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 22837
I20260929 20:52:09.459059 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-1/wal/instance
I20260929 20:52:09.459448 388 external_mini_cluster.cc:1428] Setting key 8d80cc4a924953e16fd03192f6bab641
I20260929 20:52:09.464090 22904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:52:09 dist-test-slave-blv7 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260929 20:52:09.483484 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.466596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37824 (local address 127.0.97.62:39333)
0929 20:52:09.466808 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:09.466814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:09.467947 (+ 1133us) server_negotiation.cc:407] Connection header received
0929 20:52:09.469771 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:09.469777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:09.469857 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:09.469943 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:09.471468 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.472128 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.473619 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.474029 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.474139 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:09.477704 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:09.477709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:09.477712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:09.477754 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:09.480369 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:09.481080 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:09.481084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:09.481086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:09.481235 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:09.481628 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:09.481631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:09.481633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:09.481945 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:09.482127 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:09.482915 (+ 788us) server_negotiation.cc:299] Negotiation successful
0929 20:52:09.483175 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:52:09.484472 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.467091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39333 (local address 127.0.0.1:37824)
0929 20:52:09.467795 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.467832 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:09.469496 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:09.470185 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:09.470194 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:09.470700 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:09.471303 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:09.471320 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.472276 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.472280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:09.473483 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:09.473495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.474240 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.474246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:09.474459 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:09.475159 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:09.475181 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:09.477483 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:09.480511 (+ 3028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:09.480521 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:09.480539 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:09.480957 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:09.481336 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:09.481341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:09.481345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:09.481517 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:09.482096 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:09.482104 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:09.482522 (+ 418us) client_negotiation.cc:769] Sending connection context
0929 20:52:09.482795 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 20:52:09.483380 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":269,"threads_started":1}
I20260929 20:52:09.486232 22904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:52:09 dist-test-slave-blv7 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 20:52:09.499702 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.487003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44563 (local address 127.0.0.1:36218)
0929 20:52:09.487174 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.487188 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:09.487269 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:09.488232 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:09.488236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:09.488646 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:09.488776 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:09.488784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.490096 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.490100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:09.490908 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:09.490918 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.491681 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.491687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:09.491869 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:09.492412 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:09.492428 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:09.494420 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:09.497373 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:09.497378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:09.497383 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:09.497730 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:09.498068 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:09.498073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:09.498077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:09.498199 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:09.498769 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:09.498773 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:09.498921 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 20:52:09.499053 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 20:52:09.499323 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 20:52:09.499782 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.487260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36218 (local address 127.0.97.61:44563)
0929 20:52:09.487846 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:09.487852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:09.487876 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:52:09.487916 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:09.487922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:09.488100 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:09.488227 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:09.488925 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.489962 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.491045 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.491559 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.491661 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:09.494612 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:09.494623 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:09.494636 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:09.494665 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:09.497237 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:09.497857 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:09.497864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:09.497866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:09.497977 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:09.498293 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:09.498298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:09.498301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:09.498644 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:09.498836 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:09.499207 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 20:52:09.499428 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":318,"threads_started":1}
I20260929 20:52:09.502265 22904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:52:09.503494 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.503048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34189 (local address 127.0.0.1:56024)
0929 20:52:09.503204 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.503323 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 20:52:09.503809 22855 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
I20260929 20:52:09.505448 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.504985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34189 (local address 127.0.0.1:56040)
0929 20:52:09.505216 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.505327 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260929 20:52:09.505672 22904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34189: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 20:52:09.511593 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34189
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39333,127.0.97.61:44563,127.0.97.60:34189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:52:09.555045 22904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } attempt: 1
I20260929 20:52:09.556185 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.555748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34189 (local address 127.0.0.1:56042)
0929 20:52:09.555936 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.556035 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 20:52:09.557212 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.556859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34189 (local address 127.0.0.1:56048)
0929 20:52:09.557021 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.557108 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 20:52:09.557416 22904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34189: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
I20260929 20:52:09.625635 22904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } attempt: 2
I20260929 20:52:09.626663 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.626239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34189 (local address 127.0.0.1:56058)
0929 20:52:09.626431 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.626514 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 20:52:09.627760 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.627439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34189 (local address 127.0.0.1:56064)
0929 20:52:09.627576 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.627644 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 20:52:09.627992 22904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34189: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
I20260929 20:52:09.671444 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } attempt: 5
Sep 29 20:52:09 dist-test-slave-blv7 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 20:52:09.685238 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.672156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44563 (local address 127.0.0.1:36230)
0929 20:52:09.672369 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.672390 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:09.672546 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:09.672941 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:09.672944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:09.673314 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:09.673476 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:09.673485 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.674800 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.674804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:09.675675 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:09.675686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.676657 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.676660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:09.676787 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:09.677416 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:09.677432 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:09.679647 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:09.683061 (+ 3414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:09.683066 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:09.683071 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:09.683433 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:09.683787 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:09.683790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:09.683793 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:09.683907 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:09.684563 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:09.684566 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:09.684694 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 20:52:09.684791 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 20:52:09.684994 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 20:52:09.685472 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.672226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36230 (local address 127.0.97.61:44563)
0929 20:52:09.672380 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:09.672385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:09.672491 (+ 106us) server_negotiation.cc:407] Connection header received
0929 20:52:09.672692 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:09.672698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:09.672803 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:09.672910 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:09.673621 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.674679 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.675822 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.676522 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.676664 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:09.679831 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:09.679838 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:09.679842 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:09.679881 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:09.682908 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:09.683539 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:09.683545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:09.683549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:09.683696 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:09.684052 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:09.684057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:09.684060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:09.684458 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:09.684603 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:09.684929 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 20:52:09.685150 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 20:52:09.686511 22830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:52:09.687747 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.687288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34189 (local address 127.0.0.1:56070)
0929 20:52:09.687487 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.687578 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 20:52:09.688764 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.688379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34189 (local address 127.0.0.1:56084)
0929 20:52:09.688536 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.688626 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 20:52:09.688975 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34189: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
I20260929 20:52:09.719172 22904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } attempt: 3
I20260929 20:52:09.720289 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.719780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34189 (local address 127.0.0.1:56086)
0929 20:52:09.719988 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.720090 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 20:52:09.721388 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.720977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34189 (local address 127.0.0.1:56098)
0929 20:52:09.721199 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.721282 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260929 20:52:09.721586 22904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34189: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
I20260929 20:52:09.740201 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } attempt: 1
I20260929 20:52:09.741439 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.740911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34189 (local address 127.0.0.1:56104)
0929 20:52:09.741128 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.741242 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 20:52:09.742434 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.742040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34189 (local address 127.0.0.1:56112)
0929 20:52:09.742205 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.742301 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 20:52:09.742652 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34189: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
I20260929 20:52:09.810883 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } attempt: 2
I20260929 20:52:09.812109 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.811582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34189 (local address 127.0.0.1:56122)
0929 20:52:09.811803 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.811918 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 20:52:09.813069 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.812695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34189 (local address 127.0.0.1:56132)
0929 20:52:09.812847 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.812928 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 20:52:09.813277 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34189: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
W20260929 20:52:09.832187 22911 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:52:09.832522 22911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:52:09.832592 22911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:52:09.832671 22911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:52:09.841079 22911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:52:09.841176 22911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:52:09.841228 22911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:52:09.841279 22911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:52:09.841336 22911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:52:09.851322 22911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39333,127.0.97.61:44563,127.0.97.60:34189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:52:09.852573 22911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:52:09.853812 22911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:52:09.864939 22917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:09.864872 22916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:52:09.864985 22904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } attempt: 4
W20260929 20:52:09.864907 22919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:52:09.866061 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.865600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34189 (local address 127.0.0.1:56140)
0929 20:52:09.865800 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.865888 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 20:52:09.866042 22911 server_base.cc:1034] running on GCE node
I20260929 20:52:09.866982 22911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:52:09.867110 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.866770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34189 (local address 127.0.0.1:56142)
0929 20:52:09.866899 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.866965 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 20:52:09.867349 22904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34189: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
I20260929 20:52:09.868489 22911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:52:09.869724 22911 hybrid_clock.cc:648] HybridClock initialized: now 1790715129869643 us; error 48 us; skew 500 ppm
I20260929 20:52:09.870062 22911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:52:09 dist-test-slave-blv7 krb5kdc[22750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:52:09.873060 22911 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 20:52:09.874446 22911 webserver.cc:460] Webserver started at http://127.0.97.60:44567/ using document root <none> and password file <none>
I20260929 20:52:09.875144 22911 fs_manager.cc:359] Metadata directory not provided
I20260929 20:52:09.875232 22911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:52:09.875494 22911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:52:09.878345 22911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-2/data/instance:
uuid: "ac0fe92fda8b4d40b8ed197a21731a5c"
format_stamp: "Formatted at 2026-09-29 20:52:09 on dist-test-slave-blv7"
server_key: "967f5ba5e20ff973b11abc19fe9f3a3e"
server_key_iv: "b2109f55dee369c6dff13f5111b0bd10"
server_key_version: "encryptionkey@0"
I20260929 20:52:09.879153 22911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-2/wal/instance:
uuid: "ac0fe92fda8b4d40b8ed197a21731a5c"
format_stamp: "Formatted at 2026-09-29 20:52:09 on dist-test-slave-blv7"
server_key: "967f5ba5e20ff973b11abc19fe9f3a3e"
server_key_iv: "b2109f55dee369c6dff13f5111b0bd10"
server_key_version: "encryptionkey@0"
I20260929 20:52:09.884182 22911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:52:09.887435 22926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:09.888697 22911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:52:09.888862 22911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-2/wal
uuid: "ac0fe92fda8b4d40b8ed197a21731a5c"
format_stamp: "Formatted at 2026-09-29 20:52:09 on dist-test-slave-blv7"
server_key: "967f5ba5e20ff973b11abc19fe9f3a3e"
server_key_iv: "b2109f55dee369c6dff13f5111b0bd10"
server_key_version: "encryptionkey@0"
I20260929 20:52:09.889050 22911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:52:09.907369 22911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:52:09.911137 22911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:52:09.911370 22911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:52:09.919476 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } attempt: 3
I20260929 20:52:09.920698 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.920091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34189 (local address 127.0.0.1:56158)
0929 20:52:09.920376 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.920495 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 20:52:09.921942 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.921471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34189 (local address 127.0.0.1:56170)
0929 20:52:09.921690 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.921818 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260929 20:52:09.922135 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34189: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34189: connect: Connection refused (error 111)
I20260929 20:52:09.929176 22911 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:34189
I20260929 20:52:09.929196 22977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:34189 every 8 connection(s)
I20260929 20:52:09.930532 22911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-2/data/info.pb
I20260929 20:52:09.933940 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 22911
I20260929 20:52:09.934134 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790714786636870-388-0/minicluster-data/master-2/wal/instance
I20260929 20:52:09.934610 22978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:52:09.934677 388 external_mini_cluster.cc:1428] Setting key bc55718fc825d3599b309633d4b51014
I20260929 20:52:09.944291 22978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:52:09 dist-test-slave-blv7 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 20:52:09 dist-test-slave-blv7 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 20:52:09.957835 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.936536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56186 (local address 127.0.97.60:34189)
0929 20:52:09.937299 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:09.937311 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:09.937355 (+ 44us) server_negotiation.cc:407] Connection header received
0929 20:52:09.939396 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:09.939420 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:09.939843 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:09.940177 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:09.941949 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.943296 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.945020 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.945731 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.945859 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:09.950183 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:09.950190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:09.950200 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:09.950231 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:09.953510 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:09.954187 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:09.954193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:09.954200 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:09.954376 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:09.954769 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:09.954774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:09.954776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:09.955383 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:09.955607 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:09.956066 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 20:52:09.956509 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":338,"threads_started":1}
I20260929 20:52:09.961380 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.947138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39333 (local address 127.0.0.1:37832)
0929 20:52:09.947912 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.947969 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:09.948135 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:09.948475 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:09.948483 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:09.949095 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:09.949363 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:09.949372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.950534 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.950538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:09.952158 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:09.952172 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.952892 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.952900 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:09.953078 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:09.953627 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:09.953648 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:09.955826 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:09.958893 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:09.958903 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:09.958910 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:09.959372 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:09.959716 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:09.959721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:09.959725 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:09.959859 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:09.960420 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:09.960426 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:09.960611 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 20:52:09.960792 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 20:52:09.961101 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":363,"threads_started":1}
I20260929 20:52:09.961545 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.947200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37832 (local address 127.0.97.62:39333)
0929 20:52:09.947441 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:09.947448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:09.948068 (+ 620us) server_negotiation.cc:407] Connection header received
0929 20:52:09.948238 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:09.948242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:09.948311 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:09.948386 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:09.949507 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.950391 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.952296 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.952710 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.952806 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:09.956002 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:09.956009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:09.956014 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:09.956051 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:09.958780 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:09.959490 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:09.959494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:09.959496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:09.959617 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:09.959958 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:09.959961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:09.959963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:09.960286 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:09.960451 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:09.960943 (+ 492us) server_negotiation.cc:299] Negotiation successful
0929 20:52:09.961176 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 20:52:09.962791 22978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:52:09 dist-test-slave-blv7 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 29 20:52:09 dist-test-slave-blv7 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 20:52:09.974866 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.963385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44563 (local address 127.0.0.1:36246)
0929 20:52:09.963524 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.963541 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:09.963621 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:09.963931 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:09.963934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:09.964223 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:09.964371 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:09.964378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.965294 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.965299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:09.966121 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:09.966131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.966924 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.966930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:09.967128 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:09.967574 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:09.967589 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:09.969800 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:09.972602 (+ 2802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:09.972608 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:09.972612 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:09.972943 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:09.973281 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:09.973286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:09.973290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:09.973418 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:09.973973 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:09.973978 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:09.974132 (+ 154us) client_negotiation.cc:769] Sending connection context
0929 20:52:09.974283 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 20:52:09.974535 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 20:52:09.974949 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.963493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36246 (local address 127.0.97.61:44563)
0929 20:52:09.963670 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:09.963675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:09.963693 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:52:09.963748 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:09.963753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:09.963828 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:09.963922 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:09.964509 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.965155 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.966243 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.966788 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.966892 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:09.969962 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:09.969967 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:09.969970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:09.970001 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:09.972485 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:09.973047 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:09.973051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:09.973053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:09.973174 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:09.973511 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:09.973516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:09.973519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:09.973838 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:09.973991 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:09.974387 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 20:52:09.974609 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:52:09.976191 22978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:52:09.976372 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.960629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37834 (local address 127.0.97.62:39333)
0929 20:52:09.961334 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:09.961339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:09.961361 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:52:09.961410 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:09.961415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:09.961745 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:09.961844 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:09.963188 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.964226 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.965375 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.965780 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.965885 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:09.969801 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:09.969807 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:09.969812 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:09.969848 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:09.973545 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:09.974245 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:09.974250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:09.974253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:09.974413 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:09.974848 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:09.974853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:09.974855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:09.975213 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:09.975471 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:09.975743 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 20:52:09.975964 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":381,"threads_started":1}
Sep 29 20:52:09 dist-test-slave-blv7 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 20:52:09 dist-test-slave-blv7 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 20:52:09.989667 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.977315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56190 (local address 127.0.97.60:34189)
0929 20:52:09.977453 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:09.977456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:09.977469 (+ 13us) server_negotiation.cc:407] Connection header received
0929 20:52:09.977503 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:09.977506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:09.977556 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:09.977647 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:09.978409 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.979451 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.980883 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.981444 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.981566 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:09.984715 (+ 3149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:09.984722 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:09.984727 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:09.984762 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:09.987295 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:09.988002 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:09.988006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:09.988008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:09.988111 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:09.988453 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:09.988455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:09.988457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:09.988781 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:09.988883 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:09.989262 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 20:52:09.989426 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 20:52:09.990007 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.976914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34189 (local address 127.0.0.1:56190)
0929 20:52:09.977061 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:09.977077 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:09.977168 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:09.977675 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:09.977679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:09.978113 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:09.978277 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:09.978287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.979563 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.979566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:09.980756 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:09.980769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.981589 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.981593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:09.981739 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:09.982234 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:09.982263 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:09.984543 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:09.987422 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:09.987427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:09.987431 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:09.987890 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:09.988204 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:09.988209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:09.988213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:09.988360 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:09.988886 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:09.988891 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:09.989036 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 20:52:09.989193 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 20:52:09.989626 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":111}
I20260929 20:52:09.992666 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:09.978101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36262 (local address 127.0.97.61:44563)
0929 20:52:09.978253 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:09.978257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:09.978269 (+ 12us) server_negotiation.cc:407] Connection header received
0929 20:52:09.978305 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:09.978308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:09.978360 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:09.978440 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:09.979926 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.980935 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.982245 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:09.982864 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:09.982979 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:09.987145 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:09.987150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:09.987153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:09.987182 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:09.990091 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:09.990822 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:09.990826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:09.990828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:09.990946 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:09.991384 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:09.991390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:09.991394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:09.991639 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:09.991900 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:09.992152 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 20:52:09.992375 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:52:09.995290 22978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac0fe92fda8b4d40b8ed197a21731a5c: Bootstrap starting.
I20260929 20:52:09.996927 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:52:09.998652 22978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac0fe92fda8b4d40b8ed197a21731a5c: Neither blocks nor log segments found. Creating new log.
I20260929 20:52:09.999747 22978 log.cc:824] T 00000000000000000000000000000000 P ac0fe92fda8b4d40b8ed197a21731a5c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:52:09.999747 22805 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7afe332fb7a04dd29a499c00fd678eb7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37834
I20260929 20:52:10.002307 22978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac0fe92fda8b4d40b8ed197a21731a5c: No bootstrap required, opened a new log
I20260929 20:52:10.009413 22978 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac0fe92fda8b4d40b8ed197a21731a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7afe332fb7a04dd29a499c00fd678eb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } } peers { permanent_uuid: "ef7fec96f2a7441d90cc966b38b9b511" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } } peers { permanent_uuid: "ac0fe92fda8b4d40b8ed197a21731a5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } }
I20260929 20:52:10.009757 22978 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac0fe92fda8b4d40b8ed197a21731a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac0fe92fda8b4d40b8ed197a21731a5c, State: Initialized, Role: FOLLOWER
I20260929 20:52:10.010179 22978 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac0fe92fda8b4d40b8ed197a21731a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7afe332fb7a04dd29a499c00fd678eb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } } peers { permanent_uuid: "ef7fec96f2a7441d90cc966b38b9b511" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } } peers { permanent_uuid: "ac0fe92fda8b4d40b8ed197a21731a5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } }
I20260929 20:52:10.011124 22985 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac0fe92fda8b4d40b8ed197a21731a5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7afe332fb7a04dd29a499c00fd678eb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } } peers { permanent_uuid: "ef7fec96f2a7441d90cc966b38b9b511" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } } peers { permanent_uuid: "ac0fe92fda8b4d40b8ed197a21731a5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } } }
I20260929 20:52:10.011513 22985 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac0fe92fda8b4d40b8ed197a21731a5c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:52:10.012180 22978 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac0fe92fda8b4d40b8ed197a21731a5c [sys.catalog]: configured and running, proceeding with master startup.
W20260929 20:52:10.020514 22996 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac0fe92fda8b4d40b8ed197a21731a5c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:52:10.020645 22996 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:52:10.019075 22911 master_runner.cc:186] Error getting master registration for 127.0.97.62:39333: OK, instance_id { permanent_uuid: "7afe332fb7a04dd29a499c00fd678eb7" instance_seqno: 1790715128963902 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:52:10.029824 22911 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:52:10.099296 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } attempt: 4
Sep 29 20:52:10 dist-test-slave-blv7 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 20:52:10.116202 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:10.100082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34189 (local address 127.0.0.1:56206)
0929 20:52:10.100309 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:10.100331 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:10.100455 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:10.100791 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:10.100796 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:10.101287 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:10.101510 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:10.101522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.102973 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.102979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:10.104518 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:10.104533 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.105599 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.105605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:10.105788 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:10.106444 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:10.106467 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:10.109352 (+ 2885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:10.113387 (+ 4035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:10.113393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:10.113398 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:10.113941 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:10.114309 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:10.114314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:10.114318 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:10.114487 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:10.115151 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:10.115157 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:10.115339 (+ 182us) client_negotiation.cc:769] Sending connection context
0929 20:52:10.115492 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 20:52:10.115790 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 20:52:10.116214 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:10.100121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56206 (local address 127.0.97.60:34189)
0929 20:52:10.100449 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.100455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.100483 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:52:10.100573 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.100578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.100656 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.100758 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:10.101672 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.102827 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.104694 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.105454 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.105576 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:10.109571 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:10.109578 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:10.109583 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:10.109628 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:10.113230 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:10.114044 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:10.114049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:10.114053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:10.114206 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:10.114585 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:10.114590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:10.114593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:10.115034 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:10.115178 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:10.115634 (+ 456us) server_negotiation.cc:299] Negotiation successful
0929 20:52:10.115853 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260929 20:52:10.120574 22830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7: Bootstrap starting.
I20260929 20:52:10.123653 22830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7: Neither blocks nor log segments found. Creating new log.
I20260929 20:52:10.124634 22830 log.cc:824] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7: Log is configured to *not* fsync() on all Append() calls
I20260929 20:52:10.127316 22830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7: No bootstrap required, opened a new log
I20260929 20:52:10.132171 22830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7afe332fb7a04dd29a499c00fd678eb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } } peers { permanent_uuid: "ef7fec96f2a7441d90cc966b38b9b511" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } } peers { permanent_uuid: "ac0fe92fda8b4d40b8ed197a21731a5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } }
I20260929 20:52:10.132486 22830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7afe332fb7a04dd29a499c00fd678eb7, State: Initialized, Role: FOLLOWER
I20260929 20:52:10.132884 22830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7afe332fb7a04dd29a499c00fd678eb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } } peers { permanent_uuid: "ef7fec96f2a7441d90cc966b38b9b511" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } } peers { permanent_uuid: "ac0fe92fda8b4d40b8ed197a21731a5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } }
I20260929 20:52:10.133795 22999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7afe332fb7a04dd29a499c00fd678eb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } } peers { permanent_uuid: "ef7fec96f2a7441d90cc966b38b9b511" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } } peers { permanent_uuid: "ac0fe92fda8b4d40b8ed197a21731a5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } } }
I20260929 20:52:10.134241 22999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:52:10.134945 22830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:52:10.135087 22805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 20:52:10.135296 22805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:52:10.138706 22805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7afe332fb7a04dd29a499c00fd678eb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } } peers { permanent_uuid: "ef7fec96f2a7441d90cc966b38b9b511" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } } peers { permanent_uuid: "ac0fe92fda8b4d40b8ed197a21731a5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } }
I20260929 20:52:10.140725 22805 leader_election.cc:288] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7 [CANDIDATE]: Term 1 election: Requested vote from peers ef7fec96f2a7441d90cc966b38b9b511 (127.0.97.61:44563), ac0fe92fda8b4d40b8ed197a21731a5c (127.0.97.60:34189)
I20260929 20:52:10.141394 22953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7afe332fb7a04dd29a499c00fd678eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ac0fe92fda8b4d40b8ed197a21731a5c"
I20260929 20:52:10.141924 22953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac0fe92fda8b4d40b8ed197a21731a5c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:52:10.145337 22953 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ac0fe92fda8b4d40b8ed197a21731a5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7afe332fb7a04dd29a499c00fd678eb7 in term 1.
I20260929 20:52:10.145887 22781 leader_election.cc:302] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7afe332fb7a04dd29a499c00fd678eb7, ac0fe92fda8b4d40b8ed197a21731a5c; no voters:
I20260929 20:52:10.146438 22999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:52:10.151259 22999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7 [term 1 LEADER]: Becoming Leader. State: Replica: 7afe332fb7a04dd29a499c00fd678eb7, State: Running, Role: LEADER
I20260929 20:52:10.151806 22999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7afe332fb7a04dd29a499c00fd678eb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } } peers { permanent_uuid: "ef7fec96f2a7441d90cc966b38b9b511" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } } peers { permanent_uuid: "ac0fe92fda8b4d40b8ed197a21731a5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } }
W20260929 20:52:10.163378 22932 tablet.cc:2369] T 00000000000000000000000000000000 P ac0fe92fda8b4d40b8ed197a21731a5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:52:10.166523 22904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } attempt: 5
I20260929 20:52:10.167385 23012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7afe332fb7a04dd29a499c00fd678eb7. Latest consensus state: current_term: 1 leader_uuid: "7afe332fb7a04dd29a499c00fd678eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7afe332fb7a04dd29a499c00fd678eb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } } peers { permanent_uuid: "ef7fec96f2a7441d90cc966b38b9b511" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } } peers { permanent_uuid: "ac0fe92fda8b4d40b8ed197a21731a5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } } }
I20260929 20:52:10.167867 23012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7 [sys.catalog]: This master's current role is: LEADER
I20260929 20:52:10.178475 23016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:52:10.187311 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:10.145409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37844 (local address 127.0.97.62:39333)
0929 20:52:10.145591 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.145597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.155693 (+ 10096us) server_negotiation.cc:407] Connection header received
0929 20:52:10.156327 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.156333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.156431 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.156499 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:10.160562 (+ 4063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.161701 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.172386 (+ 10685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.173160 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.173309 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:10.177325 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:10.177332 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:10.177338 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:10.177390 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:10.181316 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:10.184758 (+ 3442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:10.184765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:10.184768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:10.184941 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:10.185389 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:10.185394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:10.185397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:10.185762 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:10.186005 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:10.186627 (+ 622us) server_negotiation.cc:299] Negotiation successful
0929 20:52:10.186882 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 20:52:10.191207 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:10.146940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36278 (local address 127.0.97.61:44563)
0929 20:52:10.147162 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.147168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.170563 (+ 23395us) server_negotiation.cc:407] Connection header received
0929 20:52:10.171091 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.171098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.171194 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.171281 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:10.173568 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.174745 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.176742 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.177385 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.177488 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:10.183712 (+ 6224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:10.183719 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:10.183724 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:10.183765 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:10.187182 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:10.188609 (+ 1427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:10.188614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:10.188617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:10.188770 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:10.189492 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:10.189497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:10.189500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:10.189857 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:10.190090 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:10.190613 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 20:52:10.190842 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 20:52:10 dist-test-slave-blv7 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 20:52:10.199344 23016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:52:10.203944 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:10.145118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.0.97.60:34189)
0929 20:52:10.145303 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.145309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.160365 (+ 15056us) server_negotiation.cc:407] Connection header received
0929 20:52:10.160520 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.160527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.160621 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.160730 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:10.164505 (+ 3775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.165697 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.178515 (+ 12818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.179474 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.179610 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:10.187122 (+ 7512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:10.187130 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:10.187135 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:10.187182 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:10.190743 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:10.201027 (+ 10284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:10.201034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:10.201039 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:10.201229 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:10.201720 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:10.201726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:10.201730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:10.202102 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:10.203191 (+ 1089us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:10.203336 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 20:52:10.203572 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 20:52:10.204574 22784 tablet.cc:2369] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:52:10.207263 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:10.167278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34189 (local address 127.0.0.1:56236)
0929 20:52:10.167679 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:10.167699 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:10.167819 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:10.182017 (+ 14198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:10.182023 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:10.182501 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:10.182705 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:10.182718 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.191130 (+ 8412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.191137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:10.192521 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:10.192536 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.193608 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.193614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:10.193813 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:10.194464 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:10.194486 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:10.198555 (+ 4069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:10.204370 (+ 5815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:10.204377 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:10.204382 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:10.204965 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:10.205411 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:10.205416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:10.205420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:10.205587 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:10.206308 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:10.206313 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:10.206470 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 20:52:10.206602 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 20:52:10.206894 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266}
I20260929 20:52:10.209628 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:10.168032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56236 (local address 127.0.97.60:34189)
0929 20:52:10.181200 (+ 13168us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.181206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.181230 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:52:10.181282 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.181286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.181358 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.182874 (+ 1516us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:10.183173 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.184281 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.192717 (+ 8436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.193464 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.193586 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:10.198916 (+ 5330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:10.198928 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:10.198933 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:10.199063 (+ 130us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:10.204190 (+ 5127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:10.205140 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:10.205147 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:10.205150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:10.205285 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:10.205726 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:10.205730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:10.205733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:10.206093 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:10.207668 (+ 1575us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:10.207807 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 20:52:10.208021 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13024,"thread_start_us":12809,"threads_started":1}
I20260929 20:52:10.215965 22904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef7fec96f2a7441d90cc966b38b9b511: Bootstrap starting.
I20260929 20:52:10.220305 22904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef7fec96f2a7441d90cc966b38b9b511: Neither blocks nor log segments found. Creating new log.
I20260929 20:52:10.221688 22904 log.cc:824] T 00000000000000000000000000000000 P ef7fec96f2a7441d90cc966b38b9b511: Log is configured to *not* fsync() on all Append() calls
I20260929 20:52:10.227054 22763 master_runner.cc:186] Error getting master registration for 127.0.97.61:44563: OK, instance_id { permanent_uuid: "ef7fec96f2a7441d90cc966b38b9b511" instance_seqno: 1790715129447808 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:52:10.241387 22763 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:52:10.243897 22904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef7fec96f2a7441d90cc966b38b9b511: No bootstrap required, opened a new log
I20260929 20:52:10.245088 22953 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ac0fe92fda8b4d40b8ed197a21731a5c [term 1 FOLLOWER]: Refusing update from remote peer 7afe332fb7a04dd29a499c00fd678eb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:52:10.246135 23012 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac0fe92fda8b4d40b8ed197a21731a5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:52:10.254554 22904 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef7fec96f2a7441d90cc966b38b9b511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7afe332fb7a04dd29a499c00fd678eb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } } peers { permanent_uuid: "ef7fec96f2a7441d90cc966b38b9b511" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } } peers { permanent_uuid: "ac0fe92fda8b4d40b8ed197a21731a5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } }
I20260929 20:52:10.255041 22904 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef7fec96f2a7441d90cc966b38b9b511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef7fec96f2a7441d90cc966b38b9b511, State: Initialized, Role: FOLLOWER
I20260929 20:52:10.255645 22904 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef7fec96f2a7441d90cc966b38b9b511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7afe332fb7a04dd29a499c00fd678eb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } } peers { permanent_uuid: "ef7fec96f2a7441d90cc966b38b9b511" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } } peers { permanent_uuid: "ac0fe92fda8b4d40b8ed197a21731a5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } }
I20260929 20:52:10.268206 22904 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef7fec96f2a7441d90cc966b38b9b511 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:52:10.268260 22879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7afe332fb7a04dd29a499c00fd678eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ef7fec96f2a7441d90cc966b38b9b511"
I20260929 20:52:10.268628 22878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef7fec96f2a7441d90cc966b38b9b511 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:52:10.272557 22878 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ef7fec96f2a7441d90cc966b38b9b511 [term 1 FOLLOWER]: Refusing update from remote peer 7afe332fb7a04dd29a499c00fd678eb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:52:10.273692 23012 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef7fec96f2a7441d90cc966b38b9b511" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:52:10.276157 22879 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ef7fec96f2a7441d90cc966b38b9b511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7afe332fb7a04dd29a499c00fd678eb7 in term 1.
I20260929 20:52:10.286403 22985 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac0fe92fda8b4d40b8ed197a21731a5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7afe332fb7a04dd29a499c00fd678eb7. Latest consensus state: current_term: 1 leader_uuid: "7afe332fb7a04dd29a499c00fd678eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7afe332fb7a04dd29a499c00fd678eb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } } peers { permanent_uuid: "ef7fec96f2a7441d90cc966b38b9b511" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } } peers { permanent_uuid: "ac0fe92fda8b4d40b8ed197a21731a5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } } }
I20260929 20:52:10.286715 22985 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac0fe92fda8b4d40b8ed197a21731a5c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:52:10.287786 23026 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef7fec96f2a7441d90cc966b38b9b511 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7afe332fb7a04dd29a499c00fd678eb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } } peers { permanent_uuid: "ef7fec96f2a7441d90cc966b38b9b511" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } } peers { permanent_uuid: "ac0fe92fda8b4d40b8ed197a21731a5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } } }
I20260929 20:52:10.288956 23026 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef7fec96f2a7441d90cc966b38b9b511 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:52:10.309712 23026 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef7fec96f2a7441d90cc966b38b9b511 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7afe332fb7a04dd29a499c00fd678eb7. Latest consensus state: current_term: 1 leader_uuid: "7afe332fb7a04dd29a499c00fd678eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7afe332fb7a04dd29a499c00fd678eb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } } peers { permanent_uuid: "ef7fec96f2a7441d90cc966b38b9b511" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } } peers { permanent_uuid: "ac0fe92fda8b4d40b8ed197a21731a5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } } }
I20260929 20:52:10.310251 23026 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef7fec96f2a7441d90cc966b38b9b511 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:52:10.310091 23012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7afe332fb7a04dd29a499c00fd678eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7afe332fb7a04dd29a499c00fd678eb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } } peers { permanent_uuid: "ef7fec96f2a7441d90cc966b38b9b511" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } } peers { permanent_uuid: "ac0fe92fda8b4d40b8ed197a21731a5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } } }
I20260929 20:52:10.310364 23012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7 [sys.catalog]: This master's current role is: LEADER
I20260929 20:52:10.314316 23012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7afe332fb7a04dd29a499c00fd678eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7afe332fb7a04dd29a499c00fd678eb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } } peers { permanent_uuid: "ef7fec96f2a7441d90cc966b38b9b511" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } } peers { permanent_uuid: "ac0fe92fda8b4d40b8ed197a21731a5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } } }
I20260929 20:52:10.314591 23012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7 [sys.catalog]: This master's current role is: LEADER
I20260929 20:52:10.315351 22985 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac0fe92fda8b4d40b8ed197a21731a5c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7afe332fb7a04dd29a499c00fd678eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7afe332fb7a04dd29a499c00fd678eb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } } peers { permanent_uuid: "ef7fec96f2a7441d90cc966b38b9b511" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } } peers { permanent_uuid: "ac0fe92fda8b4d40b8ed197a21731a5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } } }
I20260929 20:52:10.315598 22985 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac0fe92fda8b4d40b8ed197a21731a5c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:52:10.317641 23026 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef7fec96f2a7441d90cc966b38b9b511 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7afe332fb7a04dd29a499c00fd678eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7afe332fb7a04dd29a499c00fd678eb7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39333 } } peers { permanent_uuid: "ef7fec96f2a7441d90cc966b38b9b511" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44563 } } peers { permanent_uuid: "ac0fe92fda8b4d40b8ed197a21731a5c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34189 } } }
I20260929 20:52:10.317864 23026 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef7fec96f2a7441d90cc966b38b9b511 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:52:10.318913 23030 mvcc.cc:205] Tried to move back new op lower bound from 7334769173426360320 to 7334769173107826688. Current Snapshot: MvccSnapshot[applied={T|T < 7334769173426360320 or (T in {7334769173426360320})}]
I20260929 20:52:10.319648 23016 catalog_manager.cc:1347] Generated new cluster ID: 6746a535d3014ef7b14f53b70cd673f1
I20260929 20:52:10.319746 23016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:52:10.354457 23016 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:52:10.356318 23016 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:52:10.373835 23016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7afe332fb7a04dd29a499c00fd678eb7: Generated new TSK 0
I20260929 20:52:10.378683 23016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:52:10.406827 23045 catalog_manager.cc:1259] Loaded cluster ID: 6746a535d3014ef7b14f53b70cd673f1
I20260929 20:52:10.406942 23045 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ef7fec96f2a7441d90cc966b38b9b511: loading cluster ID for follower catalog manager: success
I20260929 20:52:10.412252 23045 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ef7fec96f2a7441d90cc966b38b9b511: acquiring CA information for follower catalog manager: success
I20260929 20:52:10.414693 23045 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ef7fec96f2a7441d90cc966b38b9b511: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 20:52:10.539567 22983 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:10.529417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37854 (local address 127.0.97.62:39333)
0929 20:52:10.529590 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.529596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.537525 (+ 7929us) server_negotiation.cc:407] Connection header received
0929 20:52:10.538029 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.538037 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.538124 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.538195 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:10.539295 (+ 1100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37854: BlockingRecv error: recv got EOF from 127.0.0.1:37854 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 20:52:10.541679 23017 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:10.529054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56248 (local address 127.0.97.60:34189)
0929 20:52:10.529269 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.529276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.540082 (+ 10806us) server_negotiation.cc:407] Connection header received
0929 20:52:10.540313 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.540319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.540400 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.540466 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:10.541446 (+ 980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56248: BlockingRecv error: recv got EOF from 127.0.0.1:56248 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 20:52:10.543684 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:10.528947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36294 (local address 127.0.97.61:44563)
0929 20:52:10.529147 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.529153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.542184 (+ 13031us) server_negotiation.cc:407] Connection header received
0929 20:52:10.542404 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.542410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.542520 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.542585 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:10.543466 (+ 881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36294: BlockingRecv error: recv got EOF from 127.0.0.1:36294 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:52:10.569446 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:10.563479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36300 (local address 127.0.97.61:44563)
0929 20:52:10.563676 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.563682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.564119 (+ 437us) server_negotiation.cc:407] Connection header received
0929 20:52:10.564338 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.564343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.564525 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.564655 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:52:10.565061 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.565805 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.567133 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.567578 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.567686 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:10.568157 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:52:10.568700 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:52:10.568817 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:10.568984 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 20:52:10.569075 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 20:52:10.578156 23017 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:10.576433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.0.97.60:34189)
0929 20:52:10.576625 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.576630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.576878 (+ 248us) server_negotiation.cc:407] Connection header received
0929 20:52:10.577083 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.577087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.577191 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.577280 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:10.577974 (+ 694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56256: BlockingRecv error: recv got EOF from 127.0.0.1:56256 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:52:10.592032 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:10.582337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36310 (local address 127.0.97.61:44563)
0929 20:52:10.582521 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.582528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.587561 (+ 5033us) server_negotiation.cc:407] Connection header received
0929 20:52:10.587748 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.587751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.587816 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.587894 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:52:10.588305 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.589041 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.590301 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.590771 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.590876 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:10.591304 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:52:10.591420 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:52:10.591490 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:10.591624 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 20:52:10.591702 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260929 20:52:10.600332 23017 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:10.598984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56260 (local address 127.0.97.60:34189)
0929 20:52:10.599165 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.599171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.599189 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:52:10.599255 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.599261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.599326 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.599415 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:10.600119 (+ 704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56260: BlockingRecv error: recv got EOF from 127.0.0.1:56260 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:52:10.613484 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:10.608305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36324 (local address 127.0.97.61:44563)
0929 20:52:10.608509 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.608515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.608979 (+ 464us) server_negotiation.cc:407] Connection header received
0929 20:52:10.609149 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.609155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.609222 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.609303 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:52:10.609690 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.610447 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.611729 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.612284 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.612397 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:10.612772 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:52:10.612890 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:52:10.612970 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:10.613091 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 20:52:10.613164 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 20:52:10.621011 23017 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:10.619228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56270 (local address 127.0.97.60:34189)
0929 20:52:10.619412 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.619419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.619710 (+ 291us) server_negotiation.cc:407] Connection header received
0929 20:52:10.619887 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.619893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.619973 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.620053 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:10.620790 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56270: BlockingRecv error: recv got EOF from 127.0.0.1:56270 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:52:10.632836 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:10.627617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36326 (local address 127.0.97.61:44563)
0929 20:52:10.627943 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.627950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.628421 (+ 471us) server_negotiation.cc:407] Connection header received
0929 20:52:10.628599 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.628604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.628677 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.628758 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:52:10.629150 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.629888 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.631183 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.631670 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.631789 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:10.632146 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:52:10.632243 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:52:10.632330 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:10.632464 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 20:52:10.632544 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
W20260929 20:52:10.641048 23017 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:10.638267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56280 (local address 127.0.97.60:34189)
0929 20:52:10.639107 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.639114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.639731 (+ 617us) server_negotiation.cc:407] Connection header received
0929 20:52:10.639906 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.639912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.639987 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.640069 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:10.640835 (+ 766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56280: BlockingRecv error: recv got EOF from 127.0.0.1:56280 (error 108)
Metrics: {"server-negotiator.queue_time_us":690}
I20260929 20:52:10.657994 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:10.652450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36336 (local address 127.0.97.61:44563)
0929 20:52:10.652660 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.652666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.653097 (+ 431us) server_negotiation.cc:407] Connection header received
0929 20:52:10.653284 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.653289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.653358 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.653434 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:52:10.653827 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.654546 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.656317 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.656774 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.656888 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:10.657293 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:52:10.657395 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:52:10.657467 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:10.657604 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 20:52:10.657682 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 20:52:10.665647 23017 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:10.663708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56288 (local address 127.0.97.60:34189)
0929 20:52:10.663911 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.663917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.664337 (+ 420us) server_negotiation.cc:407] Connection header received
0929 20:52:10.664494 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.664500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.664573 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.664653 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:10.665447 (+ 794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56288: BlockingRecv error: recv got EOF from 127.0.0.1:56288 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:52:10.694195 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:10.688781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36342 (local address 127.0.97.61:44563)
0929 20:52:10.688990 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.688996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.689467 (+ 471us) server_negotiation.cc:407] Connection header received
0929 20:52:10.689654 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.689659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.689736 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.689814 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:52:10.690203 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.691004 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.692344 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.692847 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.692953 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:10.693409 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:52:10.693563 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:52:10.693637 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:10.693835 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 20:52:10.693921 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 20:52:10.701630 23017 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:10.699689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56300 (local address 127.0.97.60:34189)
0929 20:52:10.699894 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.699900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.700346 (+ 446us) server_negotiation.cc:407] Connection header received
0929 20:52:10.700524 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.700530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.700605 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.700686 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:10.701401 (+ 715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56300: BlockingRecv error: recv got EOF from 127.0.0.1:56300 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 20:52:10.742404 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:10.736855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36356 (local address 127.0.97.61:44563)
0929 20:52:10.737055 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.737062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.737535 (+ 473us) server_negotiation.cc:407] Connection header received
0929 20:52:10.737769 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.737773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.737847 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.737929 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:52:10.738339 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.739131 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.740562 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.741135 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.741264 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:10.741681 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:52:10.741790 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:52:10.741865 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:10.742007 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 20:52:10.742081 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 20:52:10.749859 23017 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:10.747801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56308 (local address 127.0.97.60:34189)
0929 20:52:10.747975 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.747981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.748657 (+ 676us) server_negotiation.cc:407] Connection header received
0929 20:52:10.748821 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.748827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.748902 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.748982 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:10.749673 (+ 691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56308: BlockingRecv error: recv got EOF from 127.0.0.1:56308 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:52:10.823205 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:10.817803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36370 (local address 127.0.97.61:44563)
0929 20:52:10.818005 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.818011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.818488 (+ 477us) server_negotiation.cc:407] Connection header received
0929 20:52:10.818701 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.818705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.818776 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.818852 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:52:10.819288 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.820115 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.821405 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.821906 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.822021 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:10.822439 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:52:10.822562 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:52:10.822639 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:10.822777 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 20:52:10.822855 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 20:52:10.830669 23017 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:10.828981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56314 (local address 127.0.97.60:34189)
0929 20:52:10.829199 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.829205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.829410 (+ 205us) server_negotiation.cc:407] Connection header received
0929 20:52:10.829552 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.829558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.829667 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.829787 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:10.830458 (+ 671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56314: BlockingRecv error: recv got EOF from 127.0.0.1:56314 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 20:52:10.966506 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:10.961673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36372 (local address 127.0.97.61:44563)
0929 20:52:10.961891 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.961898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.962167 (+ 269us) server_negotiation.cc:407] Connection header received
0929 20:52:10.962360 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.962364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.962430 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.962506 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:52:10.962944 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.963635 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.964908 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:10.965350 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:10.965456 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:10.965844 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:52:10.965960 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:52:10.966033 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:10.966169 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 20:52:10.966247 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 20:52:10.974251 23017 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:10.972516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.0.97.60:34189)
0929 20:52:10.972703 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:10.972708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:10.972920 (+ 212us) server_negotiation.cc:407] Connection header received
0929 20:52:10.973077 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:10.973081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:10.973151 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:10.973250 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:10.973995 (+ 745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56322: BlockingRecv error: recv got EOF from 127.0.0.1:56322 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:52:11.022603 22996 catalog_manager.cc:1259] Loaded cluster ID: 6746a535d3014ef7b14f53b70cd673f1
I20260929 20:52:11.022696 22996 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ac0fe92fda8b4d40b8ed197a21731a5c: loading cluster ID for follower catalog manager: success
I20260929 20:52:11.025131 22996 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ac0fe92fda8b4d40b8ed197a21731a5c: acquiring CA information for follower catalog manager: success
I20260929 20:52:11.026832 22996 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ac0fe92fda8b4d40b8ed197a21731a5c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:52:11.239609 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:11.234480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36380 (local address 127.0.97.61:44563)
0929 20:52:11.234709 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:11.234715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:11.234943 (+ 228us) server_negotiation.cc:407] Connection header received
0929 20:52:11.235214 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:11.235219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:11.235304 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:11.235396 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:52:11.235986 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:11.236765 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:11.238100 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:11.238492 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:11.238632 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:11.238957 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:52:11.239088 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:52:11.239152 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:11.239235 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 20:52:11.239296 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 20:52:11.250442 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:11.245279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56338 (local address 127.0.97.60:34189)
0929 20:52:11.245497 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:11.245503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:11.245667 (+ 164us) server_negotiation.cc:407] Connection header received
0929 20:52:11.245797 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:11.245801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:11.245955 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:11.246050 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:52:11.246372 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:11.247128 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:11.248351 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:11.248824 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:11.248938 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:11.249387 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:52:11.249968 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:52:11.250072 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:11.250161 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 20:52:11.250224 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 20:52:11.265690 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:11.255899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36388 (local address 127.0.97.61:44563)
0929 20:52:11.256080 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:11.256086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:11.260084 (+ 3998us) server_negotiation.cc:407] Connection header received
0929 20:52:11.260236 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:11.260241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:11.260333 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:11.260420 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:52:11.261098 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:11.261888 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:11.264013 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:11.264436 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:11.264562 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:11.264971 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:52:11.265072 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:52:11.265150 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:11.265339 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 20:52:11.265424 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:52:11.266253 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:11.256352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56344 (local address 127.0.97.60:34189)
0929 20:52:11.256513 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:11.256519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:11.260037 (+ 3518us) server_negotiation.cc:407] Connection header received
0929 20:52:11.260380 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:11.260383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:11.260461 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:11.260539 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:52:11.261260 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:11.262350 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:11.264494 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:11.265015 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:11.265130 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:11.265556 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:52:11.265667 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:52:11.265757 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:11.265914 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 20:52:11.265992 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:52:11.266652 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:11.255902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37858 (local address 127.0.97.62:39333)
0929 20:52:11.259445 (+ 3543us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:11.259451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:11.259854 (+ 403us) server_negotiation.cc:407] Connection header received
0929 20:52:11.260090 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:11.260095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:11.260337 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:11.260502 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:52:11.261363 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:11.262424 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:11.264075 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:11.264664 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:11.264793 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:11.265277 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:52:11.265657 (+ 380us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:52:11.265771 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:11.266079 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 20:52:11.266161 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3415,"thread_start_us":3311,"threads_started":1}
I20260929 20:52:11.271680 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 22763
I20260929 20:52:11.272825 22825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:52:11.389380 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 22763
W20260929 20:52:11.405839 22855 connection.cc:535] server connection from 127.0.0.1:36230 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 20:52:11.405997 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 22837
I20260929 20:52:11.407001 22899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:52:11.512282 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 22837
I20260929 20:52:11.528543 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 22911
I20260929 20:52:11.529539 22973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:52:11.633925 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 22911
2026-09-29T20:52:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4326 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:52:11 dist-test-slave-blv7 krb5kdc[23169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:52:11 dist-test-slave-blv7 krb5kdc[23169](info): set up 2 sockets
Sep 29 20:52:11 dist-test-slave-blv7 krb5kdc[23169](info): commencing operation
krb5kdc: starting...
W20260929 20:52:12.643378 388 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/test-user.keytab.
Sep 29 20:52:12 dist-test-slave-blv7 krb5kdc[23169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:52:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:52:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 20:52:12.885303 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36065
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36065,127.0.97.61:43393,127.0.97.60:40127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:52:13.204775 23182 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:52:13.205132 23182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:52:13.205231 23182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:52:13.205358 23182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:52:13.212561 23182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:52:13.212663 23182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:52:13.212743 23182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:52:13.212810 23182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:52:13.212893 23182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:52:13.222553 23182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36065,127.0.97.61:43393,127.0.97.60:40127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:52:13.223855 23182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:52:13.225200 23182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:52:13.236158 23188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:13.236220 23190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:13.236208 23187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:52:13.237236 23182 server_base.cc:1034] running on GCE node
I20260929 20:52:13.238009 23182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:52:13.239457 23182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:52:13.240670 23182 hybrid_clock.cc:648] HybridClock initialized: now 1790715133240603 us; error 46 us; skew 500 ppm
I20260929 20:52:13.240978 23182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:52:13 dist-test-slave-blv7 krb5kdc[23169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:52:13.249454 23182 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 20:52:13.250901 23182 webserver.cc:460] Webserver started at http://127.0.97.62:33191/ using document root <none> and password file <none>
I20260929 20:52:13.251618 23182 fs_manager.cc:359] Metadata directory not provided
I20260929 20:52:13.251716 23182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:52:13.252007 23182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:52:13.254944 23182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-0/data/instance:
uuid: "45a4fea3d01540a9a9cb08c27da58999"
format_stamp: "Formatted at 2026-09-29 20:52:13 on dist-test-slave-blv7"
server_key: "b240220ad407512416b973ad6c1c893b"
server_key_iv: "9729f1c079ae17054e43533acb1e246a"
server_key_version: "encryptionkey@0"
I20260929 20:52:13.255759 23182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-0/wal/instance:
uuid: "45a4fea3d01540a9a9cb08c27da58999"
format_stamp: "Formatted at 2026-09-29 20:52:13 on dist-test-slave-blv7"
server_key: "b240220ad407512416b973ad6c1c893b"
server_key_iv: "9729f1c079ae17054e43533acb1e246a"
server_key_version: "encryptionkey@0"
I20260929 20:52:13.261091 23182 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:52:13.264654 23197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:13.265935 23182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:52:13.266106 23182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-0/wal
uuid: "45a4fea3d01540a9a9cb08c27da58999"
format_stamp: "Formatted at 2026-09-29 20:52:13 on dist-test-slave-blv7"
server_key: "b240220ad407512416b973ad6c1c893b"
server_key_iv: "9729f1c079ae17054e43533acb1e246a"
server_key_version: "encryptionkey@0"
I20260929 20:52:13.266332 23182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:52:13.308486 23182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:52:13.312485 23182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:52:13.312731 23182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:52:13.329936 23182 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36065
I20260929 20:52:13.329955 23248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36065 every 8 connection(s)
I20260929 20:52:13.331295 23182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-0/data/info.pb
I20260929 20:52:13.334793 23249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:52:13.337590 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 23182
I20260929 20:52:13.337776 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-0/wal/instance
I20260929 20:52:13.338279 388 external_mini_cluster.cc:1428] Setting key 986a0820fe2d7b0e3c9359874636a311
I20260929 20:52:13.342801 23249 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36065 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:52:13 dist-test-slave-blv7 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260929 20:52:13.376042 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:13.346504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36065 (local address 127.0.0.1:59346)
0929 20:52:13.347367 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:13.347422 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:13.349630 (+ 2208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:13.350506 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:13.350513 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:13.351299 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:13.351725 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:13.351747 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:13.353272 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:13.353278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:13.354780 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:13.354794 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:13.355715 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:13.355722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:13.355897 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:13.356505 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:13.356533 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:13.358876 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:13.361942 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:13.361953 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:13.361966 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:13.362402 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:13.362966 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:13.362972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:13.362976 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:13.363201 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:13.364298 (+ 1097us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:13.364305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:13.364545 (+ 240us) client_negotiation.cc:769] Sending connection context
0929 20:52:13.364844 (+ 299us) client_negotiation.cc:240] Negotiation successful
0929 20:52:13.365587 (+ 743us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":193,"thread_start_us":352,"threads_started":1}
I20260929 20:52:13.376032 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:13.347803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59346 (local address 127.0.97.62:36065)
0929 20:52:13.348490 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:13.348499 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:13.348562 (+ 63us) server_negotiation.cc:407] Connection header received
0929 20:52:13.350016 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:13.350022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:13.350277 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:13.350437 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:13.351945 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:13.353113 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:13.354911 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:13.355490 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:13.355621 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:13.359112 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:13.359119 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:13.359132 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:13.359162 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:13.361808 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:13.362565 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:13.362575 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:13.362578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:13.362784 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:13.363324 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:13.363329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:13.363331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:13.364035 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:13.364230 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:13.365002 (+ 772us) server_negotiation.cc:299] Negotiation successful
0929 20:52:13.365412 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":473,"thread_start_us":301,"threads_started":1}
I20260929 20:52:13.379992 23249 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:52:13.381076 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:13.380666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43393 (local address 127.0.0.1:53390)
0929 20:52:13.380841 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:13.380948 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 20:52:13.381421 23200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43393: connect: Connection refused (error 111)
I20260929 20:52:13.382973 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:13.382630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43393 (local address 127.0.0.1:53396)
0929 20:52:13.382778 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:13.382852 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 20:52:13.383173 23249 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43393: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43393: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 20:52:13.399492 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43393
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36065,127.0.97.61:43393,127.0.97.60:40127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:52:13.432471 23249 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } attempt: 1
I20260929 20:52:13.433538 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:13.433134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43393 (local address 127.0.0.1:53408)
0929 20:52:13.433303 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:13.433388 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 20:52:13.434437 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:13.434129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43393 (local address 127.0.0.1:53418)
0929 20:52:13.434253 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:13.434318 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 20:52:13.434616 23249 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43393: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43393: connect: Connection refused (error 111)
I20260929 20:52:13.502867 23249 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } attempt: 2
I20260929 20:52:13.504019 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:13.503535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43393 (local address 127.0.0.1:53424)
0929 20:52:13.503791 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:13.503881 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 20:52:13.505093 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:13.504758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43393 (local address 127.0.0.1:53426)
0929 20:52:13.504898 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:13.504975 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 20:52:13.505296 23249 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43393: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43393: connect: Connection refused (error 111)
I20260929 20:52:13.596544 23249 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } attempt: 3
I20260929 20:52:13.597687 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:13.597153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43393 (local address 127.0.0.1:53440)
0929 20:52:13.597398 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:13.597512 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 20:52:13.598812 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:13.598322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43393 (local address 127.0.0.1:53448)
0929 20:52:13.598516 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:13.598611 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 20:52:13.599054 23249 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43393: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43393: connect: Connection refused (error 111)
W20260929 20:52:13.717002 23256 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:52:13.717319 23256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:52:13.717389 23256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:52:13.717470 23256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:52:13.724722 23256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:52:13.724819 23256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:52:13.724882 23256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:52:13.724931 23256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:52:13.724990 23256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:52:13.734927 23256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36065,127.0.97.61:43393,127.0.97.60:40127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:43393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:52:13.736186 23256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:52:13.737468 23256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 20:52:13.742309 23249 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } attempt: 4
I20260929 20:52:13.743623 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:13.742940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43393 (local address 127.0.0.1:53452)
0929 20:52:13.743352 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:13.743458 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265}
I20260929 20:52:13.744652 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:13.744303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43393 (local address 127.0.0.1:53456)
0929 20:52:13.744458 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:13.744537 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 20:52:13.744850 23249 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:43393: Network error: Client connection negotiation failed: client connection to 127.0.97.61:43393: connect: Connection refused (error 111)
W20260929 20:52:13.748829 23261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:13.748790 23262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:13.748862 23264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:52:13.749729 23256 server_base.cc:1034] running on GCE node
I20260929 20:52:13.750473 23256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:52:13.751967 23256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:52:13.753183 23256 hybrid_clock.cc:648] HybridClock initialized: now 1790715133753108 us; error 42 us; skew 500 ppm
I20260929 20:52:13.753495 23256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:52:13 dist-test-slave-blv7 krb5kdc[23169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:52:13.756522 23256 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 20:52:13.758008 23256 webserver.cc:460] Webserver started at http://127.0.97.61:41389/ using document root <none> and password file <none>
I20260929 20:52:13.758702 23256 fs_manager.cc:359] Metadata directory not provided
I20260929 20:52:13.758798 23256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:52:13.759140 23256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:52:13.762138 23256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-1/data/instance:
uuid: "07ca79acac0a4db880ae5283dd15c53f"
format_stamp: "Formatted at 2026-09-29 20:52:13 on dist-test-slave-blv7"
server_key: "3065d6fce9948d5d8f821ce708d433c4"
server_key_iv: "1cfead84612dc5a2428daa0a3a3dd989"
server_key_version: "encryptionkey@0"
I20260929 20:52:13.762894 23256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-1/wal/instance:
uuid: "07ca79acac0a4db880ae5283dd15c53f"
format_stamp: "Formatted at 2026-09-29 20:52:13 on dist-test-slave-blv7"
server_key: "3065d6fce9948d5d8f821ce708d433c4"
server_key_iv: "1cfead84612dc5a2428daa0a3a3dd989"
server_key_version: "encryptionkey@0"
I20260929 20:52:13.768003 23256 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:52:13.771466 23271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:13.772706 23256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:52:13.772850 23256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-1/wal
uuid: "07ca79acac0a4db880ae5283dd15c53f"
format_stamp: "Formatted at 2026-09-29 20:52:13 on dist-test-slave-blv7"
server_key: "3065d6fce9948d5d8f821ce708d433c4"
server_key_iv: "1cfead84612dc5a2428daa0a3a3dd989"
server_key_version: "encryptionkey@0"
I20260929 20:52:13.773034 23256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:52:13.793434 23256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:52:13.797438 23256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:52:13.797691 23256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:52:13.815109 23256 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:43393
I20260929 20:52:13.815142 23322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:43393 every 8 connection(s)
I20260929 20:52:13.816429 23256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-1/data/info.pb
I20260929 20:52:13.820070 23323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:52:13.822329 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 23256
I20260929 20:52:13.822499 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-1/wal/instance
I20260929 20:52:13.823004 388 external_mini_cluster.cc:1428] Setting key 1a4ffcd6c3bea777a5a836cd22fe19ee
I20260929 20:52:13.828233 23323 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36065 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:52:13 dist-test-slave-blv7 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260929 20:52:13.848820 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:13.831323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59352 (local address 127.0.97.62:36065)
0929 20:52:13.831529 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:13.831538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:13.832907 (+ 1369us) server_negotiation.cc:407] Connection header received
0929 20:52:13.834936 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:13.834943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:13.835060 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:13.835162 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:13.836570 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:13.837473 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:13.839061 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:13.839608 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:13.839718 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:13.843200 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:13.843205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:13.843209 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:13.843245 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:13.845726 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:13.846591 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:13.846594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:13.846597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:13.846719 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:13.847207 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:13.847210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:13.847212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:13.847500 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:13.847613 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:13.848359 (+ 746us) server_negotiation.cc:299] Negotiation successful
0929 20:52:13.848530 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 20:52:13.849846 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:13.831887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36065 (local address 127.0.0.1:59352)
0929 20:52:13.832716 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:13.832776 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:13.834663 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:13.835354 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:13.835362 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:13.835864 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:13.836415 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:13.836433 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:13.837640 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:13.837644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:13.838885 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:13.838897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:13.839810 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:13.839814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:13.840011 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:13.840639 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:13.840662 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:13.842945 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:13.845867 (+ 2922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:13.845877 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:13.845896 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:13.846492 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:13.846818 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:13.846823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:13.846826 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:13.847106 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:13.847630 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:13.847640 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:13.848042 (+ 402us) client_negotiation.cc:769] Sending connection context
0929 20:52:13.848258 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 20:52:13.848647 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":339,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260929 20:52:13.852242 23323 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:52:13 dist-test-slave-blv7 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 20:52:13.866691 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:13.852999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43393 (local address 127.0.0.1:53472)
0929 20:52:13.853195 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:13.853212 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:13.853301 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:13.854225 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:13.854229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:13.854623 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:13.854745 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:13.854753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:13.856099 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:13.856104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:13.857126 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:13.857169 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:13.858000 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:13.858006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:13.858163 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:13.858690 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:13.858706 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:13.861026 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:13.864533 (+ 3507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:13.864537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:13.864540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:13.864882 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:13.865236 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:13.865239 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:13.865241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:13.865363 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:13.866016 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:13.866019 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:13.866142 (+ 123us) client_negotiation.cc:769] Sending connection context
0929 20:52:13.866251 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 20:52:13.866443 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 20:52:13.867010 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:13.853261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53472 (local address 127.0.97.61:43393)
0929 20:52:13.853843 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:13.853848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:13.853871 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:52:13.853911 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:13.853916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:13.854091 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:13.854219 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:13.854892 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:13.855962 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:13.857297 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:13.857851 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:13.857958 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:13.861244 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:13.861257 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:13.861274 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:13.861307 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:13.864396 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:13.864999 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:13.865007 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:13.865010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:13.865138 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:13.865459 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:13.865463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:13.865466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:13.865875 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:13.866070 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:13.866391 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 20:52:13.866632 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":313,"threads_started":1}
I20260929 20:52:13.869539 23323 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:52:13.870775 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:13.870314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40127 (local address 127.0.0.1:47004)
0929 20:52:13.870497 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:13.870625 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 20:52:13.871065 23272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
I20260929 20:52:13.872678 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:13.872333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40127 (local address 127.0.0.1:47012)
0929 20:52:13.872473 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:13.872557 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 20:52:13.872879 23323 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40127: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 20:52:13.882232 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
/tmp/dist-test-taskf3T6XB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40127
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36065,127.0.97.61:43393,127.0.97.60:40127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:52:13.922196 23323 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } attempt: 1
I20260929 20:52:13.923295 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:13.922829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40127 (local address 127.0.0.1:47028)
0929 20:52:13.923051 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:13.923145 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 20:52:13.924295 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:13.923991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40127 (local address 127.0.0.1:47040)
0929 20:52:13.924109 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:13.924193 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 20:52:13.924510 23323 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40127: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
I20260929 20:52:13.992748 23323 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } attempt: 2
I20260929 20:52:13.993933 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:13.993465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40127 (local address 127.0.0.1:47044)
0929 20:52:13.993668 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:13.993770 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 20:52:13.994940 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:13.994575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40127 (local address 127.0.0.1:47058)
0929 20:52:13.994737 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:13.994810 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 20:52:13.995157 23323 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40127: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
I20260929 20:52:14.044097 23249 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } attempt: 5
Sep 29 20:52:14 dist-test-slave-blv7 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 20:52:14.056890 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.044924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53484 (local address 127.0.97.61:43393)
0929 20:52:14.045115 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.045120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.045143 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:52:14.045364 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.045367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.045444 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.045535 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.046296 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.047070 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.048213 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.048647 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.048754 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:14.052054 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:14.052059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:14.052062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:14.052095 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:14.054527 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.055141 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.055147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.055149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.055262 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.055598 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.055603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.055606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.055871 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:14.055999 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:14.056358 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 20:52:14.056566 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 20:52:14.056888 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.044846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43393 (local address 127.0.0.1:53484)
0929 20:52:14.045065 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:14.045086 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:14.045236 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:14.045573 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:14.045577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:14.045964 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:14.046148 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:14.046158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.047214 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.047218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:14.048065 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:14.048077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.048786 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.048792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:14.048979 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:14.049562 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:14.049579 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:14.051864 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:14.054646 (+ 2782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:14.054651 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:14.054656 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:14.054999 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:14.055363 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:14.055368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:14.055372 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:14.055501 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:14.055980 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:14.055985 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:14.056127 (+ 142us) client_negotiation.cc:769] Sending connection context
0929 20:52:14.056250 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 20:52:14.056550 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 20:52:14.058148 23249 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:52:14.059167 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.058770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40127 (local address 127.0.0.1:47068)
0929 20:52:14.058928 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:14.058996 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 20:52:14.060003 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.059699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40127 (local address 127.0.0.1:47070)
0929 20:52:14.059839 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:14.059905 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260929 20:52:14.060182 23249 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40127: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
I20260929 20:52:14.086374 23323 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } attempt: 3
I20260929 20:52:14.087548 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.087064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40127 (local address 127.0.0.1:47082)
0929 20:52:14.087297 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:14.087386 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 20:52:14.088603 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.088155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40127 (local address 127.0.0.1:47090)
0929 20:52:14.088370 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:14.088443 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260929 20:52:14.088788 23323 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40127: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
I20260929 20:52:14.111416 23249 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } attempt: 1
I20260929 20:52:14.112489 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.112044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40127 (local address 127.0.0.1:47100)
0929 20:52:14.112243 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:14.112329 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 20:52:14.113426 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.113097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40127 (local address 127.0.0.1:47110)
0929 20:52:14.113246 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:14.113315 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 20:52:14.113619 23249 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40127: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
I20260929 20:52:14.181835 23249 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } attempt: 2
I20260929 20:52:14.183058 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.182511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40127 (local address 127.0.0.1:47112)
0929 20:52:14.182776 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:14.182876 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 20:52:14.183955 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.183647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40127 (local address 127.0.0.1:47120)
0929 20:52:14.183788 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:14.183854 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 20:52:14.184168 23249 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40127: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
W20260929 20:52:14.198721 23330 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:52:14.199103 23330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:52:14.199198 23330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:52:14.199306 23330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:52:14.206610 23330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:52:14.206718 23330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:52:14.206801 23330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:52:14.206866 23330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:52:14.206954 23330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:52:14.216868 23330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36065,127.0.97.61:43393,127.0.97.60:40127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 19:57:29 UTC on ccycloud.quasar-xxilbq.root.comops.site
build id 4681269
ASAN enabled
I20260929 20:52:14.218161 23330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:52:14.219442 23330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:52:14.230548 23338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:14.230587 23335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:52:14.230636 23336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:52:14.231343 23330 server_base.cc:1034] running on GCE node
I20260929 20:52:14.231994 23323 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } attempt: 4
I20260929 20:52:14.232112 23330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:52:14.233023 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.232544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40127 (local address 127.0.0.1:47126)
0929 20:52:14.232791 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:14.232871 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 20:52:14.233731 23330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:52:14.234086 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.233719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40127 (local address 127.0.0.1:47138)
0929 20:52:14.233874 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:14.233959 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 20:52:14.234345 23323 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40127: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40127: connect: Connection refused (error 111)
I20260929 20:52:14.234999 23330 hybrid_clock.cc:648] HybridClock initialized: now 1790715134234912 us; error 53 us; skew 500 ppm
I20260929 20:52:14.235401 23330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:52:14 dist-test-slave-blv7 krb5kdc[23169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:52:14.238495 23330 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 20:52:14.240093 23330 webserver.cc:460] Webserver started at http://127.0.97.60:37447/ using document root <none> and password file <none>
I20260929 20:52:14.240821 23330 fs_manager.cc:359] Metadata directory not provided
I20260929 20:52:14.240921 23330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:52:14.241250 23330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:52:14.244185 23330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-2/data/instance:
uuid: "e810499154494694ba16587c60b35788"
format_stamp: "Formatted at 2026-09-29 20:52:14 on dist-test-slave-blv7"
server_key: "b3cbbf124a92e17c0be3c42207909ada"
server_key_iv: "243d07cf1bb5971fd96eea7839f903f5"
server_key_version: "encryptionkey@0"
I20260929 20:52:14.244907 23330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-2/wal/instance:
uuid: "e810499154494694ba16587c60b35788"
format_stamp: "Formatted at 2026-09-29 20:52:14 on dist-test-slave-blv7"
server_key: "b3cbbf124a92e17c0be3c42207909ada"
server_key_iv: "243d07cf1bb5971fd96eea7839f903f5"
server_key_version: "encryptionkey@0"
I20260929 20:52:14.249814 23330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:52:14.253175 23345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:52:14.254515 23330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:52:14.254689 23330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-2/wal
uuid: "e810499154494694ba16587c60b35788"
format_stamp: "Formatted at 2026-09-29 20:52:14 on dist-test-slave-blv7"
server_key: "b3cbbf124a92e17c0be3c42207909ada"
server_key_iv: "243d07cf1bb5971fd96eea7839f903f5"
server_key_version: "encryptionkey@0"
I20260929 20:52:14.254892 23330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:52:14.268337 23330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:52:14.272325 23330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:52:14.272560 23330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:52:14.288861 23330 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:40127
I20260929 20:52:14.288880 23396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:40127 every 8 connection(s)
I20260929 20:52:14.290143 23330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-2/data/info.pb
I20260929 20:52:14.290355 23249 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } attempt: 3
I20260929 20:52:14.294127 23397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:52:14.294859 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu as pid 23330
I20260929 20:52:14.295058 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskf3T6XB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790714786636870-388-0/minicluster-data/master-2/wal/instance
I20260929 20:52:14.295607 388 external_mini_cluster.cc:1428] Setting key 99e1953860b8cb5621c9ee082dbab0f0
Sep 29 20:52:14 dist-test-slave-blv7 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 20:52:14.303500 23397 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36065 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:52:14.313090 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.290943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40127 (local address 127.0.0.1:47154)
0929 20:52:14.291244 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:14.291267 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:14.291389 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:14.295487 (+ 4098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:14.295493 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:14.296002 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:14.296213 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:14.296227 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.298672 (+ 2445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.298679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:14.300088 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:14.300105 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.301540 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.301547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:14.301737 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:14.302421 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:14.302444 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:14.305130 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:14.309925 (+ 4795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:14.309930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:14.309934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:14.310429 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:14.311047 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:14.311052 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:14.311056 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:14.311196 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:14.312041 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:14.312046 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:14.312240 (+ 194us) client_negotiation.cc:769] Sending connection context
0929 20:52:14.312380 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 20:52:14.312660 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260929 20:52:14.314795 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.292036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.0.97.60:40127)
0929 20:52:14.292795 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.292807 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.292848 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:52:14.294439 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.294486 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.295049 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.295492 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.296432 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.298484 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.300317 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.301370 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.302108 (+ 738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:14.305450 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:14.305460 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:14.305478 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:14.305525 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:14.309778 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.310596 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.310609 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.310617 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.310907 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.311310 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.311316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.311319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.311905 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:14.312104 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:14.312668 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 20:52:14.313117 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":347,"threads_started":1}
Sep 29 20:52:14 dist-test-slave-blv7 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 29 20:52:14 dist-test-slave-blv7 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 20:52:14.322348 23249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999: Bootstrap starting.
I20260929 20:52:14.324293 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.306570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36065 (local address 127.0.0.1:59354)
0929 20:52:14.307717 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:14.307763 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:14.308859 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:14.309391 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:14.309398 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:14.310146 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:14.310386 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:14.310395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.311788 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.311793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:14.313325 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:14.313338 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.314138 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.314144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:14.314449 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:14.315221 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:14.315242 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:14.317929 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:14.321501 (+ 3572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:14.321514 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:14.321521 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:14.322033 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:14.322381 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:14.322386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:14.322390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:14.322579 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:14.323158 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:14.323178 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:14.323349 (+ 171us) client_negotiation.cc:769] Sending connection context
0929 20:52:14.323547 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 20:52:14.323861 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"mutex_wait_us":40,"thread_start_us":766,"threads_started":1}
I20260929 20:52:14.324292 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.306926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59354 (local address 127.0.97.62:36065)
0929 20:52:14.307117 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.307123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.307842 (+ 719us) server_negotiation.cc:407] Connection header received
0929 20:52:14.309014 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.309019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.309097 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.309191 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.310533 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.311627 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.313508 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.313953 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.314058 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:14.318148 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:14.318156 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:14.318160 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:14.318198 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:14.321365 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.322139 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.322144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.322148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.322276 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.322686 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.322691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.322695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.323003 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:14.323188 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:14.323683 (+ 495us) server_negotiation.cc:299] Negotiation successful
0929 20:52:14.323896 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:52:14.326032 23397 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:52:14.326321 23249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999: Neither blocks nor log segments found. Creating new log.
I20260929 20:52:14.327240 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.296452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47164 (local address 127.0.97.60:40127)
0929 20:52:14.308903 (+ 12451us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.308908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.308928 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:52:14.308970 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.308975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.309035 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.309122 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.310560 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.311942 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.313584 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.314199 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.314328 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:14.319682 (+ 5354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:14.319689 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:14.319694 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:14.319729 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:14.323378 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.324181 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.324186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.324189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.324326 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.325514 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.325519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.325522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.325854 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:14.326157 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:14.326660 (+ 503us) server_negotiation.cc:299] Negotiation successful
0929 20:52:14.326843 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":12336,"thread_start_us":12149,"threads_started":1}
I20260929 20:52:14.327634 23249 log.cc:824] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999: Log is configured to *not* fsync() on all Append() calls
I20260929 20:52:14.332158 23249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999: No bootstrap required, opened a new log
Sep 29 20:52:14 dist-test-slave-blv7 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 29 20:52:14 dist-test-slave-blv7 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 20:52:14.342985 23249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4fea3d01540a9a9cb08c27da58999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36065 } } peers { permanent_uuid: "07ca79acac0a4db880ae5283dd15c53f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } } peers { permanent_uuid: "e810499154494694ba16587c60b35788" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } }
I20260929 20:52:14.343366 23249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a4fea3d01540a9a9cb08c27da58999, State: Initialized, Role: FOLLOWER
I20260929 20:52:14.343817 23249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4fea3d01540a9a9cb08c27da58999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36065 } } peers { permanent_uuid: "07ca79acac0a4db880ae5283dd15c53f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } } peers { permanent_uuid: "e810499154494694ba16587c60b35788" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } }
I20260929 20:52:14.344712 23404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4fea3d01540a9a9cb08c27da58999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36065 } } peers { permanent_uuid: "07ca79acac0a4db880ae5283dd15c53f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } } peers { permanent_uuid: "e810499154494694ba16587c60b35788" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } } }
I20260929 20:52:14.345165 23404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:52:14.345958 23249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:52:14.346959 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.329713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.0.97.62:36065)
0929 20:52:14.329870 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.329875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.329892 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:52:14.329938 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.329943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.330013 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.330107 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.331527 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.332717 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.334431 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.335206 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.335317 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:14.339735 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:14.339742 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:14.339746 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:14.339784 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:14.343817 (+ 4033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.344986 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.344991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.344994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.345126 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.345537 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.345541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.345544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.345894 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:14.346144 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:14.346444 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 20:52:14.346644 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:52:14.347537 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.327200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53486 (local address 127.0.97.61:43393)
0929 20:52:14.327393 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.327400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.327418 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:52:14.327470 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.327475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.327548 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.327640 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.328494 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.329484 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.335127 (+ 5643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.335774 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.335893 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:14.341192 (+ 5299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:14.341200 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:14.341205 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:14.341247 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:14.344405 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.345186 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.345192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.345195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.345358 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.345831 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.345836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.345839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.346246 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:14.346393 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:14.346888 (+ 495us) server_negotiation.cc:299] Negotiation successful
0929 20:52:14.347138 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:52:14.348043 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.326813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:43393 (local address 127.0.0.1:53486)
0929 20:52:14.326983 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:14.327001 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:14.327111 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:14.327688 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:14.327693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:14.328103 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:14.328333 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:14.328346 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.333868 (+ 5522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.333872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:14.334959 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:14.334974 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.337695 (+ 2721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.337707 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:14.337906 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:14.338479 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:14.338502 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:14.340949 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:14.344550 (+ 3601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:14.344556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:14.344559 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:14.345040 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:14.345520 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:14.345524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:14.345527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:14.345691 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:14.346458 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:14.346463 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:14.346619 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 20:52:14.347441 (+ 822us) client_negotiation.cc:240] Negotiation successful
0929 20:52:14.347729 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 20:52:14.349277 23397 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:52:14 dist-test-slave-blv7 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 20:52:14 dist-test-slave-blv7 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 20:52:14.365764 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.350023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47180 (local address 127.0.97.60:40127)
0929 20:52:14.350228 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.350233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.350456 (+ 223us) server_negotiation.cc:407] Connection header received
0929 20:52:14.350580 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.350585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.350652 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.350745 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.351666 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.352690 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.354248 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.354761 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.354872 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:14.359695 (+ 4823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:14.359701 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:14.359706 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:14.359739 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:14.362693 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.363472 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.363476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.363479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.363609 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.363968 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.363972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.363975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.364328 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:14.364460 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:14.365270 (+ 810us) server_negotiation.cc:299] Negotiation successful
0929 20:52:14.365445 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:52:14.365764 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.348635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.0.97.61:43393)
0929 20:52:14.348913 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.348919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.348934 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:52:14.348977 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.348982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.349036 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.349110 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.350853 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.351584 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.353186 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.353798 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.353912 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:14.358952 (+ 5040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:14.358958 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:14.358963 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:14.359033 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:14.362188 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.362986 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.362991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.362994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.363177 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.363600 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.363604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.363607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.363969 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:14.364196 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:14.365222 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0929 20:52:14.365436 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260929 20:52:14.366219 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.350085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40127 (local address 127.0.0.1:47180)
0929 20:52:14.350370 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:14.350387 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:14.350492 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:14.350773 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:14.350777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:14.351320 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:14.351514 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:14.351525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.352813 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.352819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:14.354109 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:14.354123 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.355089 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.355096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:14.355275 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:14.355796 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:14.355822 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:14.359522 (+ 3700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:14.362824 (+ 3302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:14.362828 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:14.362831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:14.363326 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:14.363714 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:14.363718 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:14.363721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:14.363845 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:14.364505 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:14.364509 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:14.364642 (+ 133us) client_negotiation.cc:769] Sending connection context
0929 20:52:14.364768 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 20:52:14.365919 (+ 1151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260929 20:52:14.370391 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:52:14.371901 23397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e810499154494694ba16587c60b35788: Bootstrap starting.
I20260929 20:52:14.373286 23224 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "45a4fea3d01540a9a9cb08c27da58999"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59358
I20260929 20:52:14.361459 23182 master_runner.cc:186] Error getting master registration for 127.0.97.61:43393: OK, instance_id { permanent_uuid: "07ca79acac0a4db880ae5283dd15c53f" instance_seqno: 1790715133810126 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:52:14.373725 23224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 20:52:14.373929 23224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:52:14.373962 23182 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:52:14.376178 23397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e810499154494694ba16587c60b35788: Neither blocks nor log segments found. Creating new log.
W20260929 20:52:14.376907 23415 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:52:14.377024 23415 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:52:14.377599 23397 log.cc:824] T 00000000000000000000000000000000 P e810499154494694ba16587c60b35788: Log is configured to *not* fsync() on all Append() calls
I20260929 20:52:14.377559 23224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4fea3d01540a9a9cb08c27da58999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36065 } } peers { permanent_uuid: "07ca79acac0a4db880ae5283dd15c53f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } } peers { permanent_uuid: "e810499154494694ba16587c60b35788" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } }
I20260929 20:52:14.379364 23224 leader_election.cc:288] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999 [CANDIDATE]: Term 1 election: Requested vote from peers 07ca79acac0a4db880ae5283dd15c53f (127.0.97.61:43393), e810499154494694ba16587c60b35788 (127.0.97.60:40127)
I20260929 20:52:14.381448 23397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e810499154494694ba16587c60b35788: No bootstrap required, opened a new log
I20260929 20:52:14.392087 23397 raft_consensus.cc:348] T 00000000000000000000000000000000 P e810499154494694ba16587c60b35788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4fea3d01540a9a9cb08c27da58999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36065 } } peers { permanent_uuid: "07ca79acac0a4db880ae5283dd15c53f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } } peers { permanent_uuid: "e810499154494694ba16587c60b35788" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } }
I20260929 20:52:14.392580 23397 raft_consensus.cc:729] T 00000000000000000000000000000000 P e810499154494694ba16587c60b35788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e810499154494694ba16587c60b35788, State: Initialized, Role: FOLLOWER
I20260929 20:52:14.393113 23397 consensus_queue.cc:260] T 00000000000000000000000000000000 P e810499154494694ba16587c60b35788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4fea3d01540a9a9cb08c27da58999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36065 } } peers { permanent_uuid: "07ca79acac0a4db880ae5283dd15c53f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } } peers { permanent_uuid: "e810499154494694ba16587c60b35788" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } }
I20260929 20:52:14.407563 23425 sys_catalog.cc:455] T 00000000000000000000000000000000 P e810499154494694ba16587c60b35788 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4fea3d01540a9a9cb08c27da58999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36065 } } peers { permanent_uuid: "07ca79acac0a4db880ae5283dd15c53f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } } peers { permanent_uuid: "e810499154494694ba16587c60b35788" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } } }
I20260929 20:52:14.407994 23425 sys_catalog.cc:458] T 00000000000000000000000000000000 P e810499154494694ba16587c60b35788 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:52:14.409039 23397 sys_catalog.cc:564] T 00000000000000000000000000000000 P e810499154494694ba16587c60b35788 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:52:14.409235 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.385329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59374 (local address 127.0.97.62:36065)
0929 20:52:14.385508 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.385513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.385532 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:52:14.385587 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.385592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.385670 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.385782 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.387908 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.389060 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.393044 (+ 3984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.393743 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.393843 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:14.400335 (+ 6492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:14.400342 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:14.400347 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:14.400396 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:14.403974 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.405683 (+ 1709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.405689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.405692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.405856 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.406762 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.406768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.406771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.407195 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:14.407463 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:14.408595 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0929 20:52:14.408861 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:52:14.409502 23372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45a4fea3d01540a9a9cb08c27da58999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e810499154494694ba16587c60b35788"
I20260929 20:52:14.410076 23372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e810499154494694ba16587c60b35788 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:52:14.413494 23372 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e810499154494694ba16587c60b35788 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45a4fea3d01540a9a9cb08c27da58999 in term 1.
I20260929 20:52:14.414160 23198 leader_election.cc:302] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a4fea3d01540a9a9cb08c27da58999, e810499154494694ba16587c60b35788; no voters:
I20260929 20:52:14.414680 23404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:52:14.420238 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.385473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53516 (local address 127.0.97.61:43393)
0929 20:52:14.385830 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.385836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.385851 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:52:14.385902 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.385907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.385977 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.386065 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.393024 (+ 6959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.394064 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.400011 (+ 5947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.400648 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.400759 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:14.409311 (+ 8552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:14.409318 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:14.409322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:14.409374 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:14.412956 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.414152 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.414159 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.414161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.414319 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.417042 (+ 2723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.417048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.417051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.417446 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:14.417701 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:14.419384 (+ 1683us) server_negotiation.cc:299] Negotiation successful
0929 20:52:14.419605 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260929 20:52:14.427521 23404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999 [term 1 LEADER]: Becoming Leader. State: Replica: 45a4fea3d01540a9a9cb08c27da58999, State: Running, Role: LEADER
I20260929 20:52:14.427888 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.385646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47184 (local address 127.0.97.60:40127)
0929 20:52:14.385806 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.385811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.399652 (+ 13841us) server_negotiation.cc:407] Connection header received
0929 20:52:14.401833 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.401840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.401923 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.401989 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.404529 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.405597 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.414854 (+ 9257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.415503 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.415596 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:14.419003 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:14.419009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:14.419051 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:14.419098 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:14.425082 (+ 5984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.425929 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.425936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.425939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.426067 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.426451 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.426457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.426460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.426788 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:14.427051 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:14.427345 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 20:52:14.427552 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 20:52:14.428160 23404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4fea3d01540a9a9cb08c27da58999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36065 } } peers { permanent_uuid: "07ca79acac0a4db880ae5283dd15c53f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } } peers { permanent_uuid: "e810499154494694ba16587c60b35788" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } }
I20260929 20:52:14.432595 23427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45a4fea3d01540a9a9cb08c27da58999. Latest consensus state: current_term: 1 leader_uuid: "45a4fea3d01540a9a9cb08c27da58999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4fea3d01540a9a9cb08c27da58999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36065 } } peers { permanent_uuid: "07ca79acac0a4db880ae5283dd15c53f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } } peers { permanent_uuid: "e810499154494694ba16587c60b35788" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } } }
I20260929 20:52:14.433012 23427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999 [sys.catalog]: This master's current role is: LEADER
I20260929 20:52:14.434127 23432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:52:14.437479 23432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:52:14.453637 23372 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e810499154494694ba16587c60b35788 [term 1 FOLLOWER]: Refusing update from remote peer 45a4fea3d01540a9a9cb08c27da58999: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:52:14.454655 23427 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e810499154494694ba16587c60b35788" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:52:14.469698 23425 sys_catalog.cc:455] T 00000000000000000000000000000000 P e810499154494694ba16587c60b35788 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45a4fea3d01540a9a9cb08c27da58999. Latest consensus state: current_term: 1 leader_uuid: "45a4fea3d01540a9a9cb08c27da58999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4fea3d01540a9a9cb08c27da58999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36065 } } peers { permanent_uuid: "07ca79acac0a4db880ae5283dd15c53f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } } peers { permanent_uuid: "e810499154494694ba16587c60b35788" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } } }
I20260929 20:52:14.470017 23425 sys_catalog.cc:458] T 00000000000000000000000000000000 P e810499154494694ba16587c60b35788 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:52:14.473819 23427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "45a4fea3d01540a9a9cb08c27da58999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4fea3d01540a9a9cb08c27da58999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36065 } } peers { permanent_uuid: "07ca79acac0a4db880ae5283dd15c53f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } } peers { permanent_uuid: "e810499154494694ba16587c60b35788" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } } }
I20260929 20:52:14.474097 23427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999 [sys.catalog]: This master's current role is: LEADER
I20260929 20:52:14.476280 23433 mvcc.cc:205] Tried to move back new op lower bound from 7334769190708826112 to 7334769190623293440. Current Snapshot: MvccSnapshot[applied={T|T < 7334769190708826112}]
I20260929 20:52:14.477594 23432 catalog_manager.cc:1347] Generated new cluster ID: f47191bcbb2a4e658ef3322a9a9b2e1e
I20260929 20:52:14.477705 23432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:52:14.484651 23425 sys_catalog.cc:455] T 00000000000000000000000000000000 P e810499154494694ba16587c60b35788 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "45a4fea3d01540a9a9cb08c27da58999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4fea3d01540a9a9cb08c27da58999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36065 } } peers { permanent_uuid: "07ca79acac0a4db880ae5283dd15c53f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } } peers { permanent_uuid: "e810499154494694ba16587c60b35788" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } } }
I20260929 20:52:14.484907 23425 sys_catalog.cc:458] T 00000000000000000000000000000000 P e810499154494694ba16587c60b35788 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:52:14.504881 23444 catalog_manager.cc:1259] Loaded cluster ID: f47191bcbb2a4e658ef3322a9a9b2e1e
I20260929 20:52:14.504987 23444 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e810499154494694ba16587c60b35788: loading cluster ID for follower catalog manager: success
I20260929 20:52:14.505406 23432 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:52:14.507103 23432 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:52:14.508653 23444 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e810499154494694ba16587c60b35788: acquiring CA information for follower catalog manager: success
W20260929 20:52:14.509902 23444 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260929 20:52:14.498009 23330 master_runner.cc:186] Error getting master registration for 127.0.97.61:43393: OK, instance_id { permanent_uuid: "07ca79acac0a4db880ae5283dd15c53f" instance_seqno: 1790715133810126 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:52:14.515654 23330 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:52:14.519291 23432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999: Generated new TSK 0
I20260929 20:52:14.521749 23432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:52:14.533545 23323 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } attempt: 5
Sep 29 20:52:14 dist-test-slave-blv7 krb5kdc[23169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790715133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 20:52:14.560882 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.534297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47200 (local address 127.0.97.60:40127)
0929 20:52:14.534464 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.534470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.534615 (+ 145us) server_negotiation.cc:407] Connection header received
0929 20:52:14.534764 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.534770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.534846 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.534937 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.536063 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.537207 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.544651 (+ 7444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.545294 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.545419 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:14.549506 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:52:14.549513 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:52:14.549517 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:52:14.549553 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:52:14.552854 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.554002 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.554008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.554037 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.557388 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:52:14.557897 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:52:14.557902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:52:14.557906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:52:14.558272 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:52:14.558399 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:14.559349 (+ 950us) server_negotiation.cc:299] Negotiation successful
0929 20:52:14.559752 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:52:14.562191 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:14.534244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40127 (local address 127.0.0.1:47200)
0929 20:52:14.534505 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 20:52:14.534524 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:52:14.534654 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:52:14.535155 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:52:14.535160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:52:14.535660 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:52:14.535876 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:14.535889 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.537386 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.537392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:14.538770 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:52:14.538784 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:52:14.545650 (+ 6866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:52:14.545657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:52:14.545839 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:14.546406 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:52:14.546428 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:52:14.549267 (+ 2839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:52:14.553046 (+ 3779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:14.553052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:14.553056 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:14.553819 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:14.557539 (+ 3720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:52:14.557544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:52:14.557548 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:52:14.557772 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:52:14.558428 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:52:14.558433 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:52:14.558843 (+ 410us) client_negotiation.cc:769] Sending connection context
0929 20:52:14.560136 (+ 1293us) client_negotiation.cc:240] Negotiation successful
0929 20:52:14.561342 (+ 1206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260929 20:52:14.573318 23323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07ca79acac0a4db880ae5283dd15c53f: Bootstrap starting.
I20260929 20:52:14.579872 23323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07ca79acac0a4db880ae5283dd15c53f: Neither blocks nor log segments found. Creating new log.
I20260929 20:52:14.582134 23323 log.cc:824] T 00000000000000000000000000000000 P 07ca79acac0a4db880ae5283dd15c53f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:52:14.596603 23323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07ca79acac0a4db880ae5283dd15c53f: No bootstrap required, opened a new log
I20260929 20:52:14.613598 23323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07ca79acac0a4db880ae5283dd15c53f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4fea3d01540a9a9cb08c27da58999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36065 } } peers { permanent_uuid: "07ca79acac0a4db880ae5283dd15c53f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } } peers { permanent_uuid: "e810499154494694ba16587c60b35788" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } }
I20260929 20:52:14.614276 23323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07ca79acac0a4db880ae5283dd15c53f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07ca79acac0a4db880ae5283dd15c53f, State: Initialized, Role: FOLLOWER
I20260929 20:52:14.615182 23323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07ca79acac0a4db880ae5283dd15c53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4fea3d01540a9a9cb08c27da58999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36065 } } peers { permanent_uuid: "07ca79acac0a4db880ae5283dd15c53f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } } peers { permanent_uuid: "e810499154494694ba16587c60b35788" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } }
I20260929 20:52:14.634032 23297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07ca79acac0a4db880ae5283dd15c53f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:52:14.638002 23297 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 07ca79acac0a4db880ae5283dd15c53f [term 1 FOLLOWER]: Refusing update from remote peer 45a4fea3d01540a9a9cb08c27da58999: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:52:14.638675 23298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45a4fea3d01540a9a9cb08c27da58999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "07ca79acac0a4db880ae5283dd15c53f"
I20260929 20:52:14.639139 23427 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07ca79acac0a4db880ae5283dd15c53f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:52:14.641974 23298 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 07ca79acac0a4db880ae5283dd15c53f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45a4fea3d01540a9a9cb08c27da58999 in term 1.
I20260929 20:52:14.642464 23323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07ca79acac0a4db880ae5283dd15c53f [sys.catalog]: configured and running, proceeding with master startup.
W20260929 20:52:14.705418 23401 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:14.683037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47204 (local address 127.0.97.60:40127)
0929 20:52:14.683250 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.683256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.699429 (+ 16173us) server_negotiation.cc:407] Connection header received
0929 20:52:14.703612 (+ 4183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.703623 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.703722 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.703805 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.705116 (+ 1311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47204: BlockingRecv error: recv got EOF from 127.0.0.1:47204 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 20:52:14.707396 23253 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:14.682906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59380 (local address 127.0.97.62:36065)
0929 20:52:14.683230 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.683237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.698868 (+ 15631us) server_negotiation.cc:407] Connection header received
0929 20:52:14.705784 (+ 6916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.705791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.705910 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.705987 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.707114 (+ 1127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59380: BlockingRecv error: recv got EOF from 127.0.0.1:59380 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 20:52:14.710857 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:14.686129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.0.97.61:43393)
0929 20:52:14.686312 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.686318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.708527 (+ 22209us) server_negotiation.cc:407] Connection header received
0929 20:52:14.709173 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.709180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.709291 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.709375 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.710620 (+ 1245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53520: BlockingRecv error: recv got EOF from 127.0.0.1:53520 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:52:14.716785 23449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07ca79acac0a4db880ae5283dd15c53f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45a4fea3d01540a9a9cb08c27da58999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4fea3d01540a9a9cb08c27da58999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36065 } } peers { permanent_uuid: "07ca79acac0a4db880ae5283dd15c53f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } } peers { permanent_uuid: "e810499154494694ba16587c60b35788" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } } }
I20260929 20:52:14.716825 23452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07ca79acac0a4db880ae5283dd15c53f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45a4fea3d01540a9a9cb08c27da58999. Latest consensus state: current_term: 1 leader_uuid: "45a4fea3d01540a9a9cb08c27da58999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4fea3d01540a9a9cb08c27da58999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36065 } } peers { permanent_uuid: "07ca79acac0a4db880ae5283dd15c53f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } } peers { permanent_uuid: "e810499154494694ba16587c60b35788" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } } }
I20260929 20:52:14.717368 23449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07ca79acac0a4db880ae5283dd15c53f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:52:14.717445 23452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07ca79acac0a4db880ae5283dd15c53f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:52:14.717727 23462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07ca79acac0a4db880ae5283dd15c53f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "45a4fea3d01540a9a9cb08c27da58999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4fea3d01540a9a9cb08c27da58999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36065 } } peers { permanent_uuid: "07ca79acac0a4db880ae5283dd15c53f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } } peers { permanent_uuid: "e810499154494694ba16587c60b35788" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } } }
I20260929 20:52:14.718132 23462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07ca79acac0a4db880ae5283dd15c53f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:52:14.727615 23475 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260929 20:52:14.739933 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:14.737600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53530 (local address 127.0.97.61:43393)
0929 20:52:14.737762 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.737769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.738504 (+ 735us) server_negotiation.cc:407] Connection header received
0929 20:52:14.738697 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.738703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.738810 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.738897 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.739736 (+ 839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53530: BlockingRecv error: recv got EOF from 127.0.0.1:53530 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 20:52:14.754493 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:14.748904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53536 (local address 127.0.97.61:43393)
0929 20:52:14.749049 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.749053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.753235 (+ 4182us) server_negotiation.cc:407] Connection header received
0929 20:52:14.753434 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.753438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.753512 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.753615 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.754333 (+ 718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53536: BlockingRecv error: recv got EOF from 127.0.0.1:53536 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260929 20:52:14.761940 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:14.760494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.0.97.61:43393)
0929 20:52:14.760658 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.760663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.760828 (+ 165us) server_negotiation.cc:407] Connection header received
0929 20:52:14.760932 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.760938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.761027 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.761109 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.761764 (+ 655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53548: BlockingRecv error: recv got EOF from 127.0.0.1:53548 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 20:52:14.771411 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:14.769207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53558 (local address 127.0.97.61:43393)
0929 20:52:14.769390 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.769397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.769993 (+ 596us) server_negotiation.cc:407] Connection header received
0929 20:52:14.770228 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.770232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.770299 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.770380 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.771177 (+ 797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53558: BlockingRecv error: recv got EOF from 127.0.0.1:53558 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 20:52:14.784082 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:14.782138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.0.97.61:43393)
0929 20:52:14.782338 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.782345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.782793 (+ 448us) server_negotiation.cc:407] Connection header received
0929 20:52:14.782978 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.782983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.783087 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.783170 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.783859 (+ 689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53564: BlockingRecv error: recv got EOF from 127.0.0.1:53564 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 20:52:14.806877 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:14.804769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53570 (local address 127.0.97.61:43393)
0929 20:52:14.804988 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.804995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.805518 (+ 523us) server_negotiation.cc:407] Connection header received
0929 20:52:14.805711 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.805722 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.805809 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.805894 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.806644 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53570: BlockingRecv error: recv got EOF from 127.0.0.1:53570 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 20:52:14.844333 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:14.842082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53578 (local address 127.0.97.61:43393)
0929 20:52:14.842280 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.842287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.842846 (+ 559us) server_negotiation.cc:407] Connection header received
0929 20:52:14.843098 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.843102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.843176 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.843262 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.844078 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53578: BlockingRecv error: recv got EOF from 127.0.0.1:53578 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 20:52:14.913934 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:14.911827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53584 (local address 127.0.97.61:43393)
0929 20:52:14.912059 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:14.912067 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:14.912591 (+ 524us) server_negotiation.cc:407] Connection header received
0929 20:52:14.912780 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:14.912784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:14.912865 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:14.912951 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:14.913726 (+ 775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53584: BlockingRecv error: recv got EOF from 127.0.0.1:53584 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 20:52:15.047328 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:15.045479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53596 (local address 127.0.97.61:43393)
0929 20:52:15.045657 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:15.045661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:15.046014 (+ 353us) server_negotiation.cc:407] Connection header received
0929 20:52:15.046176 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:15.046181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:15.046275 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:15.046377 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:15.047122 (+ 745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53596: BlockingRecv error: recv got EOF from 127.0.0.1:53596 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:52:15.130357 23427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "45a4fea3d01540a9a9cb08c27da58999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45a4fea3d01540a9a9cb08c27da58999" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36065 } } peers { permanent_uuid: "07ca79acac0a4db880ae5283dd15c53f" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 43393 } } peers { permanent_uuid: "e810499154494694ba16587c60b35788" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40127 } } }
I20260929 20:52:15.130709 23427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a4fea3d01540a9a9cb08c27da58999 [sys.catalog]: This master's current role is: LEADER
W20260929 20:52:15.310283 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:52:15.308474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53598 (local address 127.0.97.61:43393)
0929 20:52:15.308708 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:15.308715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:15.309003 (+ 288us) server_negotiation.cc:407] Connection header received
0929 20:52:15.309205 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:15.309209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:15.309282 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:15.309368 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:52:15.310078 (+ 710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53598: BlockingRecv error: recv got EOF from 127.0.0.1:53598 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 20:52:15.512564 23444 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e810499154494694ba16587c60b35788: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:52:15.731834 23475 catalog_manager.cc:1259] Loaded cluster ID: f47191bcbb2a4e658ef3322a9a9b2e1e
I20260929 20:52:15.731961 23475 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 07ca79acac0a4db880ae5283dd15c53f: loading cluster ID for follower catalog manager: success
I20260929 20:52:15.734479 23475 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 07ca79acac0a4db880ae5283dd15c53f: acquiring CA information for follower catalog manager: success
I20260929 20:52:15.736261 23475 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 07ca79acac0a4db880ae5283dd15c53f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:52:15.837898 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:15.830207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53604 (local address 127.0.97.61:43393)
0929 20:52:15.831531 (+ 1324us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:15.831538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:15.831799 (+ 261us) server_negotiation.cc:407] Connection header received
0929 20:52:15.832004 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:15.832008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:15.832151 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:15.832271 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:52:15.832731 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:15.833500 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:15.835515 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:15.835962 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:15.836048 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:15.836674 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:52:15.837284 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:52:15.837391 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:15.837528 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 20:52:15.837598 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1182,"thread_start_us":972,"threads_started":1}
I20260929 20:52:15.849738 23540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:15.844154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47206 (local address 127.0.97.60:40127)
0929 20:52:15.844789 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:15.844795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:15.844812 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:52:15.844906 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:15.844910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:15.845056 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:15.845193 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:52:15.845594 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:15.846373 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:15.847761 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:15.848127 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:15.848231 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:15.848688 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:52:15.849175 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:52:15.849304 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:15.849384 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 20:52:15.849459 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":401,"threads_started":1}
I20260929 20:52:15.875734 23540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:15.855535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47210 (local address 127.0.97.60:40127)
0929 20:52:15.862635 (+ 7100us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:15.862641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:15.868966 (+ 6325us) server_negotiation.cc:407] Connection header received
0929 20:52:15.869362 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:15.869368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:15.869460 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:15.869522 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:52:15.870336 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:15.871496 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:15.873585 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:15.874196 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:15.874280 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:15.874885 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:52:15.875054 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:52:15.875116 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:15.875355 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 20:52:15.875436 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6960}
I20260929 20:52:15.878288 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:15.855061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59382 (local address 127.0.97.62:36065)
0929 20:52:15.868901 (+ 13840us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:15.868907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:15.868930 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:52:15.869059 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:15.869065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:15.869301 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:15.869481 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:52:15.870487 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:15.871774 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:15.874040 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:15.874696 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:15.874803 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:15.876094 (+ 1291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:52:15.876639 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:52:15.876749 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:15.877922 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0929 20:52:15.878003 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13699,"thread_start_us":14733,"threads_started":1}
I20260929 20:52:15.881345 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:52:15.862300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53608 (local address 127.0.97.61:43393)
0929 20:52:15.862496 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:52:15.862503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:52:15.875855 (+ 13352us) server_negotiation.cc:407] Connection header received
0929 20:52:15.875980 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:52:15.875984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:52:15.876064 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:52:15.876170 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:52:15.876804 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:15.877902 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:15.879522 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:52:15.880020 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:52:15.880133 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:52:15.880598 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:52:15.880730 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:52:15.880821 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 20:52:15.880960 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 20:52:15.881040 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:52:15.882375 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 23182
I20260929 20:52:15.883297 23244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:52:15.989197 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 23182
I20260929 20:52:16.006280 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 23256
I20260929 20:52:16.007395 23318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:52:16.117401 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 23256
I20260929 20:52:16.132668 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu pid 23330
I20260929 20:52:16.133734 23392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:52:16.238011 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskf3T6XB/build/asan/bin/kudu with pid 23330
2026-09-29T20:52:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4603 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8930 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (349562 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST